2024-03-28T18:17:01.673 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T18:17:01.673 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386 --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/orig.config.yaml 2024-03-28T18:17:01.719 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T18:17:01.793 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} email: mchangir@redhat.com first_in_suite: false job_id: '7625386' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdoIz05eUlhUkEGcz+QQ2ZG7iE+g47sCRa3agonGiyd8To3h80ZtG4I48icaCSjSQNzbVXZ1aJDta5boLBLCD4= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOR0ZjGzjYhg8OQczKBEPQpnNAGAz+5HvUx6kV/hR3LIHpH4bLjAxlPDXfSkxYWgE323Q4hi0BGyLOhGRswGYFc= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF9yVmsF8k5npS9xLS17jnLwNvuiscQ44RKd07u+ol1/iBOvFPmrvJBEXon5VQ2EZ3fpMsfbmSnfjkhI4YUIrh4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T18:17:01.793 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T18:17:01.794 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T18:17:01.794 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T18:17:01.795 INFO:teuthology.task.internal:Checking packages... 2024-03-28T18:17:01.816 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T18:17:01.816 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T18:17:01.816 INFO:teuthology.packaging:ref: None 2024-03-28T18:17:01.817 INFO:teuthology.packaging:tag: None 2024-03-28T18:17:01.817 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T18:17:01.817 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:17:01.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T18:17:01.960 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T18:17:01.961 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T18:17:01.969 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T18:17:01.970 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T18:17:01.996 INFO:teuthology.task.internal:Saving configuration 2024-03-28T18:17:02.013 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T18:17:02.023 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:17:02.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.002316', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdoIz05eUlhUkEGcz+QQ2ZG7iE+g47sCRa3agonGiyd8To3h80ZtG4I48icaCSjSQNzbVXZ1aJDta5boLBLCD4='} 2024-03-28T18:17:02.064 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.003841', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOR0ZjGzjYhg8OQczKBEPQpnNAGAz+5HvUx6kV/hR3LIHpH4bLjAxlPDXfSkxYWgE323Q4hi0BGyLOhGRswGYFc='} 2024-03-28T18:17:02.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.005079', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF9yVmsF8k5npS9xLS17jnLwNvuiscQ44RKd07u+ol1/iBOvFPmrvJBEXon5VQ2EZ3fpMsfbmSnfjkhI4YUIrh4='} 2024-03-28T18:17:02.089 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T18:17:02.099 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T18:17:02.100 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T18:17:02.100 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T18:17:02.100 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T18:17:02.198 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faac22d7f70>, signals=[15]) 2024-03-28T18:17:02.199 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T18:17:02.207 INFO:teuthology.task.internal:Opening connections... 2024-03-28T18:17:02.207 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:17:02.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:17:02.299 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-03-28T18:17:02.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:17:02.385 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-03-28T18:17:02.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:17:02.471 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T18:17:02.478 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-03-28T18:17:02.497 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-03-28T18:17:02.497 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:17:02.554 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:17:02.555 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-03-28T18:17:02.573 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-03-28T18:17:02.591 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-03-28T18:17:02.591 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:17:02.648 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:17:02.649 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-03-28T18:17:02.673 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-03-28T18:17:02.691 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-03-28T18:17:02.691 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:17:02.748 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:17:02.749 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:17:02.749 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-03-28T18:17:02.764 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T18:17:02.776 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T18:17:02.783 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T18:17:02.783 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:17:02.786 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:17:02.788 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:17:02.804 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T18:17:02.810 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T18:17:02.811 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:17:02.842 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:17:02.844 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:17:02.924 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T18:17:02.998 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T18:17:03.011 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T18:17:03.011 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T18:17:03.011 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T18:17:03.012 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:17:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:17:03.028 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-28T18:17:03.085 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-361.el9.x86_64 2024-03-28T18:17:03.085 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-361.el9.x86_64 2024-03-28T18:17:03.085 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-03-28T18:17:05.258 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T18:17:06.859 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-28T18:17:07.641 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-28T18:17:09.475 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-28T18:17:13.551 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-28T18:17:15.768 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-28T18:17:17.105 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-28T18:17:21.969 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-28T18:17:24.109 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T18:17:24.109 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T18:17:24.210 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T18:17:24.218 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 76 M 2024-03-28T18:17:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 123 M 2024-03-28T18:17:24.220 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-28T18:17:24.454 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 554 kB/s | 66 kB 00:00 2024-03-28T18:17:25.396 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-28T18:17:26.654 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-28T18:17:27.171 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-28T18:17:27.171 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:17:27.171 INFO:teuthology.orchestra.run.smithi145.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-28T18:17:27.837 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-28T18:17:28.211 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-28T18:17:28.211 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-28T18:17:29.213 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-28T18:17:29.213 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-28T18:17:31.701 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-28T18:17:32.337 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T18:17:32.421 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T18:17:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T18:17:35.291 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T18:17:39.963 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T18:17:40.008 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:17:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:18:13.873 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:18:13.873 INFO:teuthology.orchestra.run.smithi145.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T18:18:13.873 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:18:13.917 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:18:14.278 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:18:14.278 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T18:18:14.278 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T18:18:14.279 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T18:18:14.700 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T18:18:14.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:18:14.701 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-28T18:18:15.033 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T18:18:15.132 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T18:18:15.132 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T18:18:15.132 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T18:18:15.133 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T18:18:15.133 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T18:18:15.133 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T18:18:15.133 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T18:18:15.133 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T18:18:15.213 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T18:18:15.214 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T18:18:15.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 68.el9 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8264254 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T18:18:15.249 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-03-28T18:18:15.250 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:18:15.251 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T18:18:15.252 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T18:18:15.253 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-432.el9.x86_64 2024-03-28T18:18:15.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T18:18:15.866 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-03-28T18:18:17.179 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T18:18:17.218 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-03-28T18:18:17.221 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-28T18:18:17.236 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.mEQeeiabLj 2024-03-28T18:18:17.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mEQeeiabLj 2024-03-28T18:18:17.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.mEQeeiabLj 2024-03-28T18:18:17.443 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.mEQeeiabLj is 6KB 2024-03-28T18:18:17.463 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.mEQeeiabLj 2024-03-28T18:18:17.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T18:18:17.545 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T18:18:17.545 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T18:18:17.545 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T18:18:17.545 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T18:18:17.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T18:18:17.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-03-28T18:18:47.723 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T18:18:47.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:18:47.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:06.143 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-28T18:19:15.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:19:15.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:18.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-28T18:19:30.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:19:30.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:48.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-28T18:20:03.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:20:03.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:20:04.000 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-28T18:20:04.385 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:20:04.385 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-28T18:20:04.386 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-28T18:20:04.399 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-432.el9.x86_64 2024-03-28T18:20:04.399 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T18:20:04.399 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T18:20:04.399 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T18:20:05.400 DEBUG:teuthology.parallel:result is None 2024-03-28T18:20:05.401 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T18:20:05.409 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T18:20:05.409 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:20:05.412 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:20:05.414 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:20:05.432 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T18:20:05.437 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T18:20:05.446 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T18:20:05.446 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:20:05.466 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:20:05.471 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:20:05.594 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T18:20:05.600 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T18:20:05.601 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:20:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:20:05.617 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:20:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:20:05.633 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:20:05.636 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:20:05.660 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:20:05.684 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.713 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.723 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.736 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.744 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.770 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:20:05.771 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T18:20:05.778 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T18:20:05.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:20:05.780 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:20:05.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:20:05.812 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T18:20:05.820 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T18:20:05.820 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:20:05.848 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:20:05.853 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:20:05.868 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:05.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:05.992 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.122 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:20:06.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:20:06.203 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:06.245 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:06.306 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.345 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.392 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:20:06.392 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:20:06.460 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:06.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:20:06.561 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:20:06.690 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:20:06.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:20:06.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-03-28T18:20:06.762 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-03-28T18:20:06.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-03-28T18:20:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:20:06.836 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:20:06.855 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:20:07.309 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T18:20:07.319 INFO:teuthology.task.internal:Starting timer... 2024-03-28T18:20:07.319 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T18:20:07.345 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T18:20:07.361 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T18:20:07.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-03-28T18:20:07.556 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-03-28T18:20:07.558 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-03-28T18:20:07.614 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-03-28T18:20:07.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-03-28T18:20:07.679 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-03-28T18:20:07.681 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T18:20:07.681 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-03-28T18:20:07.791 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-03-28T18:20:07.792 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-03-28T18:20:07.824 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-03-28T18:20:07.824 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-03-28T18:20:07.847 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-03-28T18:20:07.848 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi145.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-03-28T18:20:07.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:20:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:20:07.879 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:20:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:20:07.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:20:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:20:07.937 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T18:20:07.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:20:08.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:20:08.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:20:08.075 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T18:20:08.110 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T18:20:08.116 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T18:20:08.278 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T18:20:08.292 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T18:20:08.293 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi145.front.sepia.ceph.com,smithi164.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T18:24:45.692 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi145.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-03-28T18:24:45.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:24:45.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:24:45.783 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-28T18:24:45.864 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-28T18:24:45.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-28T18:24:45.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:24:45.950 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-28T18:24:46.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-28T18:24:46.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-28T18:24:46.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:24:46.102 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-28T18:24:46.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-28T18:24:46.169 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T18:24:46.180 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T18:24:46.180 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:24:46.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:24:46.183 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:24:46.183 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:24:46.185 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:24:46.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:24:46.212 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:24:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:24:46.228 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:24:46.231 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:24:46.242 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:24:46.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-03-28T18:24:46.258 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:24:46.260 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-03-28T18:24:46.271 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-03-28T18:24:46.274 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-03-28T18:24:46.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-03-28T18:24:46.288 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:24:46.290 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:24:46.301 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-03-28T18:24:46.304 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:24:46.307 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:24:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:24:46.333 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:24:46.351 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.356 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.360 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.377 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:24:46.380 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:24:46.380 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-28T18:24:46.380 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.380 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.380 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.381 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:24:46.381 INFO:teuthology.run_tasks:Running task install... 2024-03-28T18:24:46.390 DEBUG:teuthology.task.install:project ceph 2024-03-28T18:24:46.390 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T18:24:46.390 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T18:24:46.390 INFO:teuthology.task.install:Using flavor: default 2024-03-28T18:24:46.398 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T18:24:46.398 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T18:24:46.398 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:24:46.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:24:46.401 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:24:46.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:24:46.402 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:24:46.402 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:24:46.549 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:24:46.549 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:24:46.623 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:24:46.623 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:24:46.627 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:24:46.627 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:24:46.670 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:24:46.671 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:24:46.671 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:24:46.699 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:24:46.699 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:24:46.734 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:24:46.734 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:24:46.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:24:46.762 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:24:46.762 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:24:46.767 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:24:46.792 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:24:46.792 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:24:46.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:24:46.820 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:24:46.821 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:24:46.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:24:46.847 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:24:46.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:24:46.908 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:24:46.912 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-03-28T18:24:46.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-03-28T18:24:46.968 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:24:46.974 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-03-28T18:24:46.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-03-28T18:24:47.034 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2024-03-28T18:24:47.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2024-03-28T18:24:47.147 INFO:teuthology.orchestra.run.smithi164.stdout:74 files removed 2024-03-28T18:24:47.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:24:47.210 INFO:teuthology.orchestra.run.smithi192.stdout:74 files removed 2024-03-28T18:24:47.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:24:47.278 INFO:teuthology.orchestra.run.smithi145.stdout:74 files removed 2024-03-28T18:24:47.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:24:47.782 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 266 kB/s | 83 kB 00:00 2024-03-28T18:24:47.928 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-03-28T18:24:48.077 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-03-28T18:24:48.191 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-28T18:24:48.321 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-28T18:24:48.504 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-28T18:24:48.599 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-28T18:24:48.671 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-28T18:24:49.005 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-28T18:24:49.274 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-28T18:24:49.335 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-28T18:24:49.449 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-28T18:24:50.799 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-28T18:24:50.802 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-28T18:24:50.824 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-28T18:24:52.509 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-03-28T18:24:54.644 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-28T18:24:58.745 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-28T18:25:00.329 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-28T18:25:00.547 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 873 kB/s | 8.0 MB 00:09 2024-03-28T18:25:00.680 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 864 kB/s | 8.0 MB 00:09 2024-03-28T18:25:01.381 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-28T18:25:02.465 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-28T18:25:02.723 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-28T18:25:06.397 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 9.4 kB/s | 1.7 kB 00:00 2024-03-28T18:25:06.424 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-28T18:25:06.940 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-28T18:25:07.934 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-28T18:25:08.366 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-28T18:25:08.793 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:08.793 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:08.808 INFO:teuthology.orchestra.run.smithi164.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:25:08.809 INFO:teuthology.orchestra.run.smithi164.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:25:08.810 INFO:teuthology.orchestra.run.smithi164.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:25:08.812 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:08.813 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:08.817 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:25:08.818 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:25:08.899 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-28T18:25:08.918 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-28T18:25:08.918 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-28T18:25:08.918 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-28T18:25:08.918 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:25:08.919 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:25:08.920 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:25:08.921 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:25:08.922 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:25:08.923 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:25:08.924 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:25:08.925 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:25:08.926 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:25:08.927 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:25:08.928 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:25:08.929 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout:Install 150 Packages 2024-03-28T18:25:08.930 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 24 Packages 2024-03-28T18:25:08.931 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:25:08.931 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 225 M 2024-03-28T18:25:08.931 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-28T18:25:09.398 INFO:teuthology.orchestra.run.smithi164.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-28T18:25:09.573 INFO:teuthology.orchestra.run.smithi164.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 5.0 MB/s | 890 kB 00:00 2024-03-28T18:25:09.632 INFO:teuthology.orchestra.run.smithi164.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-28T18:25:09.790 INFO:teuthology.orchestra.run.smithi164.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 15 MB/s | 2.3 MB 00:00 2024-03-28T18:25:09.843 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T18:25:09.891 INFO:teuthology.orchestra.run.smithi164.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 15 MB/s | 1.5 MB 00:00 2024-03-28T18:25:09.963 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-28T18:25:10.158 INFO:teuthology.orchestra.run.smithi164.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 18 MB/s | 4.7 MB 00:00 2024-03-28T18:25:11.075 INFO:teuthology.orchestra.run.smithi164.stdout:(7/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 18 MB/s | 17 MB 00:00 2024-03-28T18:25:11.258 INFO:teuthology.orchestra.run.smithi164.stdout:(8/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.6 MB/s | 5.4 MB 00:02 2024-03-28T18:25:11.883 INFO:teuthology.orchestra.run.smithi164.stdout:(9/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 12 MB/s | 9.9 MB 00:00 2024-03-28T18:25:11.900 INFO:teuthology.orchestra.run.smithi164.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 39 kB/s | 25 kB 00:00 2024-03-28T18:25:12.375 INFO:teuthology.orchestra.run.smithi164.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 464 kB/s | 220 kB 00:00 2024-03-28T18:25:12.501 INFO:teuthology.orchestra.run.smithi164.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 216 kB/s | 27 kB 00:00 2024-03-28T18:25:12.601 INFO:teuthology.orchestra.run.smithi164.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 313 kB/s | 31 kB 00:00 2024-03-28T18:25:12.885 INFO:teuthology.orchestra.run.smithi164.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.6 MB/s | 746 kB 00:00 2024-03-28T18:25:12.993 INFO:teuthology.orchestra.run.smithi164.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.5 MB/s | 163 kB 00:00 2024-03-28T18:25:13.102 INFO:teuthology.orchestra.run.smithi164.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.1 MB/s | 127 kB 00:00 2024-03-28T18:25:13.294 INFO:teuthology.orchestra.run.smithi164.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.6 MB/s | 501 kB 00:00 2024-03-28T18:25:14.361 INFO:teuthology.orchestra.run.smithi164.stdout:(18/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 19 MB/s | 48 MB 00:02 2024-03-28T18:25:14.445 INFO:teuthology.orchestra.run.smithi164.stdout:(19/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 4.5 MB/s | 5.1 MB 00:01 2024-03-28T18:25:14.470 INFO:teuthology.orchestra.run.smithi164.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 417 kB/s | 45 kB 00:00 2024-03-28T18:25:14.520 INFO:teuthology.orchestra.run.smithi164.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 1.7 MB/s | 131 kB 00:00 2024-03-28T18:25:14.545 INFO:teuthology.orchestra.run.smithi164.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-28T18:25:14.587 INFO:teuthology.orchestra.run.smithi164.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.7 MB/s | 321 kB 00:00 2024-03-28T18:25:14.613 INFO:teuthology.orchestra.run.smithi164.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 298 kB 00:00 2024-03-28T18:25:14.663 INFO:teuthology.orchestra.run.smithi164.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T18:25:14.688 INFO:teuthology.orchestra.run.smithi164.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-28T18:25:14.738 INFO:teuthology.orchestra.run.smithi164.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T18:25:14.789 INFO:teuthology.orchestra.run.smithi164.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 678 kB/s | 34 kB 00:00 2024-03-28T18:25:14.800 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T18:25:14.856 INFO:teuthology.orchestra.run.smithi164.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-28T18:25:14.911 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-28T18:25:14.989 INFO:teuthology.orchestra.run.smithi164.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 9.5 MB/s | 3.1 MB 00:00 2024-03-28T18:25:15.031 INFO:teuthology.orchestra.run.smithi164.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 10 MB/s | 1.8 MB 00:00 2024-03-28T18:25:15.090 INFO:teuthology.orchestra.run.smithi164.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 4.2 MB/s | 248 kB 00:00 2024-03-28T18:25:15.143 INFO:teuthology.orchestra.run.smithi164.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 942 kB/s | 49 kB 00:00 2024-03-28T18:25:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 396 kB/s | 16 kB 00:00 2024-03-28T18:25:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T18:25:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 8.0 MB/s | 748 kB 00:00 2024-03-28T18:25:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 365 kB/s | 18 kB 00:00 2024-03-28T18:25:15.511 INFO:teuthology.orchestra.run.smithi164.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-03-28T18:25:15.611 INFO:teuthology.orchestra.run.smithi164.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-03-28T18:25:15.637 INFO:teuthology.orchestra.run.smithi164.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.3 MB/s | 920 kB 00:00 2024-03-28T18:25:15.662 INFO:teuthology.orchestra.run.smithi164.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-28T18:25:15.687 INFO:teuthology.orchestra.run.smithi164.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T18:25:15.713 INFO:teuthology.orchestra.run.smithi164.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-28T18:25:15.796 INFO:teuthology.orchestra.run.smithi164.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-28T18:25:15.821 INFO:teuthology.orchestra.run.smithi164.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 162 kB/s | 17 kB 00:00 2024-03-28T18:25:15.847 INFO:teuthology.orchestra.run.smithi164.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-28T18:25:15.872 INFO:teuthology.orchestra.run.smithi164.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-28T18:25:15.889 INFO:teuthology.orchestra.run.smithi164.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-28T18:25:15.914 INFO:teuthology.orchestra.run.smithi164.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-28T18:25:15.940 INFO:teuthology.orchestra.run.smithi164.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-28T18:25:15.957 INFO:teuthology.orchestra.run.smithi164.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T18:25:15.982 INFO:teuthology.orchestra.run.smithi164.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-28T18:25:16.007 INFO:teuthology.orchestra.run.smithi164.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T18:25:16.033 INFO:teuthology.orchestra.run.smithi164.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T18:25:16.049 INFO:teuthology.orchestra.run.smithi164.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-28T18:25:16.158 INFO:teuthology.orchestra.run.smithi164.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T18:25:16.192 INFO:teuthology.orchestra.run.smithi164.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-28T18:25:16.217 INFO:teuthology.orchestra.run.smithi164.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T18:25:16.351 INFO:teuthology.orchestra.run.smithi164.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-03-28T18:25:16.367 INFO:teuthology.orchestra.run.smithi164.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-28T18:25:16.384 INFO:teuthology.orchestra.run.smithi164.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 540 kB/s | 18 kB 00:00 2024-03-28T18:25:16.418 INFO:teuthology.orchestra.run.smithi164.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T18:25:16.435 INFO:teuthology.orchestra.run.smithi164.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-28T18:25:16.561 INFO:teuthology.orchestra.run.smithi164.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-28T18:25:16.586 INFO:teuthology.orchestra.run.smithi164.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T18:25:16.620 INFO:teuthology.orchestra.run.smithi164.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-28T18:25:16.645 INFO:teuthology.orchestra.run.smithi164.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-28T18:25:16.678 INFO:teuthology.orchestra.run.smithi164.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 707 kB/s | 41 kB 00:00 2024-03-28T18:25:16.712 INFO:teuthology.orchestra.run.smithi164.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-28T18:25:16.737 INFO:teuthology.orchestra.run.smithi164.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T18:25:16.771 INFO:teuthology.orchestra.run.smithi164.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-28T18:25:16.796 INFO:teuthology.orchestra.run.smithi164.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-03-28T18:25:16.821 INFO:teuthology.orchestra.run.smithi164.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-28T18:25:17.058 INFO:teuthology.orchestra.run.smithi164.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 165 kB/s | 39 kB 00:00 2024-03-28T18:25:17.178 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.187 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.193 INFO:teuthology.orchestra.run.smithi145.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:25:17.194 INFO:teuthology.orchestra.run.smithi145.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.195 INFO:teuthology.orchestra.run.smithi145.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:25:17.196 INFO:teuthology.orchestra.run.smithi145.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.198 INFO:teuthology.orchestra.run.smithi145.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.201 INFO:teuthology.orchestra.run.smithi145.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.202 INFO:teuthology.orchestra.run.smithi145.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:25:17.214 INFO:teuthology.orchestra.run.smithi164.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-28T18:25:17.231 INFO:teuthology.orchestra.run.smithi164.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 240 kB/s | 41 kB 00:00 2024-03-28T18:25:17.264 INFO:teuthology.orchestra.run.smithi164.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T18:25:17.274 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.275 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.281 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-28T18:25:17.281 INFO:teuthology.orchestra.run.smithi164.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T18:25:17.289 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:25:17.290 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:25:17.291 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:25:17.292 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.294 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.297 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:25:17.298 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:25:17.298 INFO:teuthology.orchestra.run.smithi164.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-28T18:25:17.309 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:25:17.310 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:25:17.311 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:25:17.312 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:25:17.313 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:25:17.314 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:25:17.315 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:25:17.316 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:25:17.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:25:17.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:25:17.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:25:17.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:25:17.321 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout:Install 150 Packages 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 24 Packages 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:25:17.322 INFO:teuthology.orchestra.run.smithi164.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T18:25:17.323 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 225 M 2024-03-28T18:25:17.323 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-28T18:25:17.349 INFO:teuthology.orchestra.run.smithi164.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T18:25:17.377 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-28T18:25:17.391 INFO:teuthology.orchestra.run.smithi164.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T18:25:17.393 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:25:17.394 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:25:17.395 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:25:17.396 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:25:17.397 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:25:17.398 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:25:17.399 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:25:17.400 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:25:17.401 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:25:17.402 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:25:17.403 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:25:17.404 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:25:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:Install 150 Packages 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 24 Packages 2024-03-28T18:25:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:25:17.407 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 225 M 2024-03-28T18:25:17.407 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-28T18:25:17.433 INFO:teuthology.orchestra.run.smithi164.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T18:25:17.467 INFO:teuthology.orchestra.run.smithi164.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-28T18:25:17.492 INFO:teuthology.orchestra.run.smithi164.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-28T18:25:17.517 INFO:teuthology.orchestra.run.smithi164.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-28T18:25:17.768 INFO:teuthology.orchestra.run.smithi164.stdout:(87/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.4 MB/s | 20 MB 00:08 2024-03-28T18:25:17.877 INFO:teuthology.orchestra.run.smithi164.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 488 kB/s | 187 kB 00:00 2024-03-28T18:25:17.935 INFO:teuthology.orchestra.run.smithi164.stdout:(89/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-28T18:25:17.960 INFO:teuthology.orchestra.run.smithi164.stdout:(90/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-28T18:25:18.002 INFO:teuthology.orchestra.run.smithi164.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-28T18:25:18.069 INFO:teuthology.orchestra.run.smithi164.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 161 kB/s | 21 kB 00:00 2024-03-28T18:25:18.094 INFO:teuthology.orchestra.run.smithi164.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-03-28T18:25:18.245 INFO:teuthology.orchestra.run.smithi164.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-03-28T18:25:18.303 INFO:teuthology.orchestra.run.smithi164.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-28T18:25:18.329 INFO:teuthology.orchestra.run.smithi164.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-03-28T18:25:18.352 INFO:teuthology.orchestra.run.smithi145.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-28T18:25:18.379 INFO:teuthology.orchestra.run.smithi164.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-28T18:25:18.412 INFO:teuthology.orchestra.run.smithi192.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 35 kB/s | 6.6 kB 00:00 2024-03-28T18:25:18.463 INFO:teuthology.orchestra.run.smithi164.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 9.0 MB/s | 4.1 MB 00:00 2024-03-28T18:25:18.488 INFO:teuthology.orchestra.run.smithi164.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-28T18:25:18.538 INFO:teuthology.orchestra.run.smithi164.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 718 kB/s | 36 kB 00:00 2024-03-28T18:25:18.588 INFO:teuthology.orchestra.run.smithi164.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-03-28T18:25:18.622 INFO:teuthology.orchestra.run.smithi164.stdout:(102/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-03-28T18:25:18.647 INFO:teuthology.orchestra.run.smithi164.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T18:25:18.722 INFO:teuthology.orchestra.run.smithi164.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-28T18:25:18.829 INFO:teuthology.orchestra.run.smithi192.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.1 MB/s | 890 kB 00:00 2024-03-28T18:25:18.839 INFO:teuthology.orchestra.run.smithi164.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-03-28T18:25:18.913 INFO:teuthology.orchestra.run.smithi192.stdout:(3/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.8 MB/s | 5.4 MB 00:00 2024-03-28T18:25:18.938 INFO:teuthology.orchestra.run.smithi192.stdout:(4/174): ceph-immutable-object-cache-19.0.0-214 1.3 MB/s | 145 kB 00:00 2024-03-28T18:25:18.960 INFO:teuthology.orchestra.run.smithi145.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-28T18:25:19.040 INFO:teuthology.orchestra.run.smithi164.stdout:(106/174): postgresql-server-13.14-1.el9.x86_64 7.9 MB/s | 5.8 MB 00:00 2024-03-28T18:25:19.180 INFO:teuthology.orchestra.run.smithi192.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.7 MB/s | 2.3 MB 00:00 2024-03-28T18:25:19.235 INFO:teuthology.orchestra.run.smithi145.stdout:(3/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-03-28T18:25:19.399 INFO:teuthology.orchestra.run.smithi164.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-28T18:25:19.449 INFO:teuthology.orchestra.run.smithi164.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 755 kB/s | 548 kB 00:00 2024-03-28T18:25:19.474 INFO:teuthology.orchestra.run.smithi164.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 508 kB/s | 309 kB 00:00 2024-03-28T18:25:19.480 INFO:teuthology.orchestra.run.smithi192.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-28T18:25:19.541 INFO:teuthology.orchestra.run.smithi164.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-28T18:25:19.574 INFO:teuthology.orchestra.run.smithi164.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-28T18:25:19.594 INFO:teuthology.orchestra.run.smithi145.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 6.4 MB/s | 2.3 MB 00:00 2024-03-28T18:25:19.641 INFO:teuthology.orchestra.run.smithi164.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-28T18:25:19.664 INFO:teuthology.orchestra.run.smithi192.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-28T18:25:19.708 INFO:teuthology.orchestra.run.smithi164.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-28T18:25:19.752 INFO:teuthology.orchestra.run.smithi145.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 9.5 MB/s | 1.5 MB 00:00 2024-03-28T18:25:19.778 INFO:teuthology.orchestra.run.smithi145.stdout:(6/174): ceph-immutable-object-cache-19.0.0-214 178 kB/s | 145 kB 00:00 2024-03-28T18:25:19.842 INFO:teuthology.orchestra.run.smithi164.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-28T18:25:19.901 INFO:teuthology.orchestra.run.smithi164.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-28T18:25:19.934 INFO:teuthology.orchestra.run.smithi164.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-03-28T18:25:19.993 INFO:teuthology.orchestra.run.smithi164.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-03-28T18:25:20.060 INFO:teuthology.orchestra.run.smithi164.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-03-28T18:25:20.118 INFO:teuthology.orchestra.run.smithi164.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-28T18:25:20.144 INFO:teuthology.orchestra.run.smithi164.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-28T18:25:20.169 INFO:teuthology.orchestra.run.smithi164.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-03-28T18:25:20.236 INFO:teuthology.orchestra.run.smithi164.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T18:25:20.261 INFO:teuthology.orchestra.run.smithi164.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-03-28T18:25:20.264 INFO:teuthology.orchestra.run.smithi145.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 9.2 MB/s | 4.7 MB 00:00 2024-03-28T18:25:20.286 INFO:teuthology.orchestra.run.smithi164.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-28T18:25:20.387 INFO:teuthology.orchestra.run.smithi164.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-28T18:25:20.412 INFO:teuthology.orchestra.run.smithi164.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-03-28T18:25:20.437 INFO:teuthology.orchestra.run.smithi164.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 822 kB/s | 144 kB 00:00 2024-03-28T18:25:20.487 INFO:teuthology.orchestra.run.smithi164.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-28T18:25:20.513 INFO:teuthology.orchestra.run.smithi164.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-28T18:25:20.538 INFO:teuthology.orchestra.run.smithi164.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-03-28T18:25:20.588 INFO:teuthology.orchestra.run.smithi164.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-28T18:25:20.638 INFO:teuthology.orchestra.run.smithi164.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-03-28T18:25:20.663 INFO:teuthology.orchestra.run.smithi164.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-03-28T18:25:20.689 INFO:teuthology.orchestra.run.smithi164.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-03-28T18:25:20.739 INFO:teuthology.orchestra.run.smithi164.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-28T18:25:20.781 INFO:teuthology.orchestra.run.smithi164.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-28T18:25:20.815 INFO:teuthology.orchestra.run.smithi164.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 130 kB/s | 16 kB 00:00 2024-03-28T18:25:20.840 INFO:teuthology.orchestra.run.smithi164.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-03-28T18:25:20.882 INFO:teuthology.orchestra.run.smithi164.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-28T18:25:20.916 INFO:teuthology.orchestra.run.smithi164.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-28T18:25:20.941 INFO:teuthology.orchestra.run.smithi164.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-28T18:25:20.983 INFO:teuthology.orchestra.run.smithi164.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-28T18:25:21.017 INFO:teuthology.orchestra.run.smithi164.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-28T18:25:21.042 INFO:teuthology.orchestra.run.smithi164.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-28T18:25:21.053 INFO:teuthology.orchestra.run.smithi145.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 13 MB/s | 9.9 MB 00:00 2024-03-28T18:25:21.092 INFO:teuthology.orchestra.run.smithi164.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-28T18:25:21.118 INFO:teuthology.orchestra.run.smithi164.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 893 kB/s | 90 kB 00:00 2024-03-28T18:25:21.120 INFO:teuthology.orchestra.run.smithi145.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 378 kB/s | 25 kB 00:00 2024-03-28T18:25:21.151 INFO:teuthology.orchestra.run.smithi164.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-28T18:25:21.193 INFO:teuthology.orchestra.run.smithi164.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-03-28T18:25:21.218 INFO:teuthology.orchestra.run.smithi164.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-28T18:25:21.285 INFO:teuthology.orchestra.run.smithi164.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-28T18:25:21.372 INFO:teuthology.orchestra.run.smithi192.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 5.8 MB/s | 9.9 MB 00:01 2024-03-28T18:25:21.436 INFO:teuthology.orchestra.run.smithi164.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-28T18:25:21.481 INFO:teuthology.orchestra.run.smithi192.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 233 kB/s | 25 kB 00:00 2024-03-28T18:25:22.077 INFO:teuthology.orchestra.run.smithi164.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.6 MB/s | 2.3 MB 00:00 2024-03-28T18:25:22.228 INFO:teuthology.orchestra.run.smithi164.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-03-28T18:25:22.361 INFO:teuthology.orchestra.run.smithi164.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 6.5 MB/s | 1.8 MB 00:00 2024-03-28T18:25:22.370 INFO:teuthology.orchestra.run.smithi145.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.8 MB/s | 20 MB 00:04 2024-03-28T18:25:22.386 INFO:teuthology.orchestra.run.smithi164.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 265 kB/s | 42 kB 00:00 2024-03-28T18:25:22.420 INFO:teuthology.orchestra.run.smithi164.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.1 MB/s | 248 kB 00:00 2024-03-28T18:25:22.479 INFO:teuthology.orchestra.run.smithi164.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 777 kB/s | 71 kB 00:00 2024-03-28T18:25:22.495 INFO:teuthology.orchestra.run.smithi145.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 220 kB 00:00 2024-03-28T18:25:22.496 INFO:teuthology.orchestra.run.smithi164.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 485 kB/s | 36 kB 00:00 2024-03-28T18:25:22.554 INFO:teuthology.orchestra.run.smithi164.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-28T18:25:22.579 INFO:teuthology.orchestra.run.smithi145.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 325 kB/s | 27 kB 00:00 2024-03-28T18:25:22.580 INFO:teuthology.orchestra.run.smithi164.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-28T18:25:22.605 INFO:teuthology.orchestra.run.smithi164.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-28T18:25:22.647 INFO:teuthology.orchestra.run.smithi164.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-28T18:25:22.654 INFO:teuthology.orchestra.run.smithi145.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 418 kB/s | 31 kB 00:00 2024-03-28T18:25:22.672 INFO:teuthology.orchestra.run.smithi164.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-28T18:25:22.705 INFO:teuthology.orchestra.run.smithi164.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-28T18:25:22.840 INFO:teuthology.orchestra.run.smithi164.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 381 kB/s | 64 kB 00:00 2024-03-28T18:25:22.892 INFO:teuthology.orchestra.run.smithi164.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 154 kB/s | 29 kB 00:00 2024-03-28T18:25:22.964 INFO:teuthology.orchestra.run.smithi145.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.4 MB/s | 746 kB 00:00 2024-03-28T18:25:23.017 INFO:teuthology.orchestra.run.smithi164.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 1.2 MB 00:00 2024-03-28T18:25:23.056 INFO:teuthology.orchestra.run.smithi145.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-28T18:25:23.148 INFO:teuthology.orchestra.run.smithi145.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-28T18:25:23.268 INFO:teuthology.orchestra.run.smithi164.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.1 MB/s | 2.3 MB 00:00 2024-03-28T18:25:23.306 INFO:teuthology.orchestra.run.smithi145.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.1 MB/s | 501 kB 00:00 2024-03-28T18:25:23.351 INFO:teuthology.orchestra.run.smithi164.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 2.2 MB 00:00 2024-03-28T18:25:23.377 INFO:teuthology.orchestra.run.smithi164.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.3 MB/s | 465 kB 00:00 2024-03-28T18:25:23.402 INFO:teuthology.orchestra.run.smithi164.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-28T18:25:23.502 INFO:teuthology.orchestra.run.smithi164.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-28T18:25:23.652 INFO:teuthology.orchestra.run.smithi164.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 297 kB/s | 74 kB 00:00 2024-03-28T18:25:23.861 INFO:teuthology.orchestra.run.smithi164.stdout:(174/174): librados2-19.0.0-2145.g3be9a5c9.el9. 1.3 MB/s | 3.4 MB 00:02 2024-03-28T18:25:23.865 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:25:23.865 INFO:teuthology.orchestra.run.smithi164.stdout:Total 15 MB/s | 225 MB 00:14 2024-03-28T18:25:24.452 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-28T18:25:24.565 INFO:teuthology.orchestra.run.smithi145.stdout:(18/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-28T18:25:24.665 INFO:teuthology.orchestra.run.smithi145.stdout:(19/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-03-28T18:25:24.682 INFO:teuthology.orchestra.run.smithi145.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 388 kB/s | 45 kB 00:00 2024-03-28T18:25:24.724 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-28T18:25:24.724 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-28T18:25:24.724 INFO:teuthology.orchestra.run.smithi145.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 2.2 MB/s | 131 kB 00:00 2024-03-28T18:25:24.749 INFO:teuthology.orchestra.run.smithi145.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-28T18:25:24.798 INFO:teuthology.orchestra.run.smithi192.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 3.2 MB/s | 17 MB 00:05 2024-03-28T18:25:24.816 INFO:teuthology.orchestra.run.smithi145.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 298 kB 00:00 2024-03-28T18:25:24.841 INFO:teuthology.orchestra.run.smithi145.stdout:(24/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 321 kB 00:00 2024-03-28T18:25:24.865 INFO:teuthology.orchestra.run.smithi192.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.3 MB/s | 220 kB 00:00 2024-03-28T18:25:24.875 INFO:teuthology.orchestra.run.smithi145.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 99 kB 00:00 2024-03-28T18:25:24.892 INFO:teuthology.orchestra.run.smithi145.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T18:25:24.906 INFO:teuthology.orchestra.run.smithi192.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 653 kB/s | 27 kB 00:00 2024-03-28T18:25:24.957 INFO:teuthology.orchestra.run.smithi192.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 625 kB/s | 31 kB 00:00 2024-03-28T18:25:24.959 INFO:teuthology.orchestra.run.smithi145.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-28T18:25:25.043 INFO:teuthology.orchestra.run.smithi145.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 408 kB/s | 34 kB 00:00 2024-03-28T18:25:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:(29/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 3.1 MB 00:00 2024-03-28T18:25:25.149 INFO:teuthology.orchestra.run.smithi192.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.8 MB/s | 746 kB 00:00 2024-03-28T18:25:25.168 INFO:teuthology.orchestra.run.smithi145.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.1 MB/s | 139 kB 00:00 2024-03-28T18:25:25.232 INFO:teuthology.orchestra.run.smithi192.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.9 MB/s | 163 kB 00:00 2024-03-28T18:25:25.299 INFO:teuthology.orchestra.run.smithi192.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 127 kB 00:00 2024-03-28T18:25:25.302 INFO:teuthology.orchestra.run.smithi145.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 11 MB/s | 1.8 MB 00:00 2024-03-28T18:25:25.386 INFO:teuthology.orchestra.run.smithi145.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-28T18:25:25.416 INFO:teuthology.orchestra.run.smithi192.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.2 MB/s | 501 kB 00:00 2024-03-28T18:25:25.452 INFO:teuthology.orchestra.run.smithi145.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 741 kB/s | 49 kB 00:00 2024-03-28T18:25:25.519 INFO:teuthology.orchestra.run.smithi145.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 247 kB/s | 16 kB 00:00 2024-03-28T18:25:25.586 INFO:teuthology.orchestra.run.smithi145.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 264 kB 00:00 2024-03-28T18:25:25.687 INFO:teuthology.orchestra.run.smithi145.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-28T18:25:25.753 INFO:teuthology.orchestra.run.smithi145.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 274 kB/s | 18 kB 00:00 2024-03-28T18:25:25.887 INFO:teuthology.orchestra.run.smithi145.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-28T18:25:25.938 INFO:teuthology.orchestra.run.smithi145.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-28T18:25:25.971 INFO:teuthology.orchestra.run.smithi145.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T18:25:26.014 INFO:teuthology.orchestra.run.smithi145.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T18:25:26.047 INFO:teuthology.orchestra.run.smithi145.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-28T18:25:26.147 INFO:teuthology.orchestra.run.smithi145.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T18:25:26.181 INFO:teuthology.orchestra.run.smithi145.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 516 kB/s | 17 kB 00:00 2024-03-28T18:25:26.214 INFO:teuthology.orchestra.run.smithi145.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-28T18:25:26.225 INFO:teuthology.orchestra.run.smithi192.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.4 MB/s | 5.1 MB 00:00 2024-03-28T18:25:26.248 INFO:teuthology.orchestra.run.smithi145.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-28T18:25:26.267 INFO:teuthology.orchestra.run.smithi192.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 1.1 MB/s | 45 kB 00:00 2024-03-28T18:25:26.282 INFO:teuthology.orchestra.run.smithi145.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T18:25:26.317 INFO:teuthology.orchestra.run.smithi192.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.6 MB/s | 131 kB 00:00 2024-03-28T18:25:26.332 INFO:teuthology.orchestra.run.smithi145.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T18:25:26.367 INFO:teuthology.orchestra.run.smithi192.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 3.2 MB/s | 162 kB 00:00 2024-03-28T18:25:26.382 INFO:teuthology.orchestra.run.smithi145.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-28T18:25:26.408 INFO:teuthology.orchestra.run.smithi145.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-28T18:25:26.443 INFO:teuthology.orchestra.run.smithi192.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-28T18:25:26.468 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-28T18:25:26.468 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-28T18:25:26.508 INFO:teuthology.orchestra.run.smithi145.stdout:(51/174): ceph-mgr-diskprediction-local-19.0.0- 5.5 MB/s | 7.4 MB 00:01 2024-03-28T18:25:26.518 INFO:teuthology.orchestra.run.smithi192.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T18:25:26.533 INFO:teuthology.orchestra.run.smithi145.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-28T18:25:26.559 INFO:teuthology.orchestra.run.smithi145.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T18:25:26.568 INFO:teuthology.orchestra.run.smithi192.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 99 kB 00:00 2024-03-28T18:25:26.584 INFO:teuthology.orchestra.run.smithi145.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-28T18:25:26.609 INFO:teuthology.orchestra.run.smithi145.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-28T18:25:26.618 INFO:teuthology.orchestra.run.smithi192.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T18:25:26.785 INFO:teuthology.orchestra.run.smithi145.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-28T18:25:26.818 INFO:teuthology.orchestra.run.smithi145.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-28T18:25:26.844 INFO:teuthology.orchestra.run.smithi145.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T18:25:27.002 INFO:teuthology.orchestra.run.smithi145.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-28T18:25:27.011 INFO:teuthology.orchestra.run.smithi192.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.9 MB/s | 3.1 MB 00:00 2024-03-28T18:25:27.027 INFO:teuthology.orchestra.run.smithi145.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 134 kB/s | 24 kB 00:00 2024-03-28T18:25:27.053 INFO:teuthology.orchestra.run.smithi145.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-28T18:25:27.061 INFO:teuthology.orchestra.run.smithi192.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T18:25:27.078 INFO:teuthology.orchestra.run.smithi145.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T18:25:27.103 INFO:teuthology.orchestra.run.smithi192.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 820 kB/s | 34 kB 00:00 2024-03-28T18:25:27.103 INFO:teuthology.orchestra.run.smithi145.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-28T18:25:27.153 INFO:teuthology.orchestra.run.smithi192.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T18:25:27.237 INFO:teuthology.orchestra.run.smithi145.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-28T18:25:27.262 INFO:teuthology.orchestra.run.smithi145.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T18:25:27.288 INFO:teuthology.orchestra.run.smithi145.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T18:25:27.313 INFO:teuthology.orchestra.run.smithi145.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T18:25:27.338 INFO:teuthology.orchestra.run.smithi145.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 819 kB/s | 41 kB 00:00 2024-03-28T18:25:27.363 INFO:teuthology.orchestra.run.smithi145.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T18:25:27.389 INFO:teuthology.orchestra.run.smithi145.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T18:25:27.395 INFO:teuthology.orchestra.run.smithi192.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 7.4 MB/s | 1.8 MB 00:00 2024-03-28T18:25:27.414 INFO:teuthology.orchestra.run.smithi145.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T18:25:27.456 INFO:teuthology.orchestra.run.smithi145.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 709 kB/s | 47 kB 00:00 2024-03-28T18:25:27.481 INFO:teuthology.orchestra.run.smithi145.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 642 kB/s | 43 kB 00:00 2024-03-28T18:25:27.623 INFO:teuthology.orchestra.run.smithi145.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 275 kB/s | 39 kB 00:00 2024-03-28T18:25:27.857 INFO:teuthology.orchestra.run.smithi145.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-28T18:25:27.882 INFO:teuthology.orchestra.run.smithi145.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-28T18:25:27.908 INFO:teuthology.orchestra.run.smithi145.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T18:25:27.941 INFO:teuthology.orchestra.run.smithi145.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 971 kB/s | 56 kB 00:00 2024-03-28T18:25:27.966 INFO:teuthology.orchestra.run.smithi145.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-28T18:25:27.992 INFO:teuthology.orchestra.run.smithi145.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-28T18:25:28.017 INFO:teuthology.orchestra.run.smithi145.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T18:25:28.042 INFO:teuthology.orchestra.run.smithi145.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T18:25:28.084 INFO:teuthology.orchestra.run.smithi145.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-28T18:25:28.110 INFO:teuthology.orchestra.run.smithi145.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-28T18:25:28.135 INFO:teuthology.orchestra.run.smithi145.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T18:25:28.160 INFO:teuthology.orchestra.run.smithi145.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-28T18:25:28.237 INFO:teuthology.orchestra.run.smithi192.stdout:(31/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.0 MB/s | 20 MB 00:10 2024-03-28T18:25:28.321 INFO:teuthology.orchestra.run.smithi192.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 8.0 MB/s | 7.4 MB 00:00 2024-03-28T18:25:28.371 INFO:teuthology.orchestra.run.smithi192.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 985 kB/s | 49 kB 00:00 2024-03-28T18:25:28.377 INFO:teuthology.orchestra.run.smithi145.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-28T18:25:28.396 INFO:teuthology.orchestra.run.smithi192.stdout:(34/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.5 MB/s | 248 kB 00:00 2024-03-28T18:25:28.422 INFO:teuthology.orchestra.run.smithi192.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 328 kB/s | 16 kB 00:00 2024-03-28T18:25:28.444 INFO:teuthology.orchestra.run.smithi145.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-03-28T18:25:28.478 INFO:teuthology.orchestra.run.smithi145.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-28T18:25:28.505 INFO:teuthology.orchestra.run.smithi192.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.4 MB/s | 264 kB 00:00 2024-03-28T18:25:28.519 INFO:teuthology.orchestra.run.smithi145.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T18:25:28.556 INFO:teuthology.orchestra.run.smithi192.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-28T18:25:28.581 INFO:teuthology.orchestra.run.smithi192.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 243 kB/s | 18 kB 00:00 2024-03-28T18:25:28.731 INFO:teuthology.orchestra.run.smithi192.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-28T18:25:28.737 INFO:teuthology.orchestra.run.smithi145.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 83 kB/s | 21 kB 00:00 2024-03-28T18:25:28.765 INFO:teuthology.orchestra.run.smithi192.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 920 kB 00:00 2024-03-28T18:25:28.790 INFO:teuthology.orchestra.run.smithi192.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-28T18:25:28.815 INFO:teuthology.orchestra.run.smithi192.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T18:25:28.841 INFO:teuthology.orchestra.run.smithi192.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2024-03-28T18:25:28.895 INFO:teuthology.orchestra.run.smithi145.stdout:(92/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 1.8 MB/s | 17 MB 00:09 2024-03-28T18:25:28.920 INFO:teuthology.orchestra.run.smithi145.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 63 kB/s | 25 kB 00:00 2024-03-28T18:25:29.266 INFO:teuthology.orchestra.run.smithi192.stdout:(44/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 6.1 MB/s | 48 MB 00:07 2024-03-28T18:25:29.317 INFO:teuthology.orchestra.run.smithi192.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.0 MB/s | 3.0 MB 00:00 2024-03-28T18:25:29.333 INFO:teuthology.orchestra.run.smithi192.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 17 kB 00:00 2024-03-28T18:25:29.350 INFO:teuthology.orchestra.run.smithi192.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 124 kB/s | 10 kB 00:00 2024-03-28T18:25:29.367 INFO:teuthology.orchestra.run.smithi192.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T18:25:29.393 INFO:teuthology.orchestra.run.smithi192.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-28T18:25:29.426 INFO:teuthology.orchestra.run.smithi192.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-28T18:25:29.451 INFO:teuthology.orchestra.run.smithi192.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 578 kB 00:00 2024-03-28T18:25:29.477 INFO:teuthology.orchestra.run.smithi192.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-03-28T18:25:29.494 INFO:teuthology.orchestra.run.smithi192.stdout:(53/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 702 kB/s | 46 kB 00:00 2024-03-28T18:25:29.519 INFO:teuthology.orchestra.run.smithi192.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-28T18:25:29.544 INFO:teuthology.orchestra.run.smithi192.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-28T18:25:29.561 INFO:teuthology.orchestra.run.smithi192.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 827 kB/s | 34 kB 00:00 2024-03-28T18:25:29.571 INFO:teuthology.orchestra.run.smithi145.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 2.4 MB/s | 1.6 MB 00:00 2024-03-28T18:25:29.661 INFO:teuthology.orchestra.run.smithi192.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 34 MB/s | 4.7 MB 00:00 2024-03-28T18:25:29.695 INFO:teuthology.orchestra.run.smithi192.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-28T18:25:29.696 INFO:teuthology.orchestra.run.smithi145.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 1.1 MB/s | 911 kB 00:00 2024-03-28T18:25:29.720 INFO:teuthology.orchestra.run.smithi192.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 492 kB/s | 78 kB 00:00 2024-03-28T18:25:29.721 INFO:teuthology.orchestra.run.smithi145.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 886 kB/s | 133 kB 00:00 2024-03-28T18:25:29.835 INFO:teuthology.orchestra.run.smithi192.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 5.8 MB 00:00 2024-03-28T18:25:29.838 INFO:teuthology.orchestra.run.smithi145.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.8 MB/s | 4.1 MB 00:01 2024-03-28T18:25:29.852 INFO:teuthology.orchestra.run.smithi192.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 190 kB/s | 24 kB 00:00 2024-03-28T18:25:29.864 INFO:teuthology.orchestra.run.smithi145.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-28T18:25:29.869 INFO:teuthology.orchestra.run.smithi192.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 124 kB/s | 18 kB 00:00 2024-03-28T18:25:29.894 INFO:teuthology.orchestra.run.smithi192.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-28T18:25:29.911 INFO:teuthology.orchestra.run.smithi192.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 763 kB/s | 32 kB 00:00 2024-03-28T18:25:29.972 INFO:teuthology.orchestra.run.smithi145.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-03-28T18:25:30.029 INFO:teuthology.orchestra.run.smithi192.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-28T18:25:30.053 INFO:teuthology.orchestra.run.smithi192.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T18:25:30.079 INFO:teuthology.orchestra.run.smithi192.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-28T18:25:30.081 INFO:teuthology.orchestra.run.smithi145.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 331 kB/s | 36 kB 00:00 2024-03-28T18:25:30.095 INFO:teuthology.orchestra.run.smithi192.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-28T18:25:30.106 INFO:teuthology.orchestra.run.smithi145.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 927 kB/s | 248 kB 00:00 2024-03-28T18:25:30.112 INFO:teuthology.orchestra.run.smithi192.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-28T18:25:30.137 INFO:teuthology.orchestra.run.smithi192.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-28T18:25:30.163 INFO:teuthology.orchestra.run.smithi192.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-28T18:25:30.181 INFO:teuthology.orchestra.run.smithi145.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-03-28T18:25:30.188 INFO:teuthology.orchestra.run.smithi192.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-28T18:25:30.207 INFO:teuthology.orchestra.run.smithi145.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 934 kB/s | 94 kB 00:00 2024-03-28T18:25:30.213 INFO:teuthology.orchestra.run.smithi192.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-28T18:25:30.239 INFO:teuthology.orchestra.run.smithi192.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 565 kB/s | 43 kB 00:00 2024-03-28T18:25:30.324 INFO:teuthology.orchestra.run.smithi145.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-28T18:25:30.457 INFO:teuthology.orchestra.run.smithi145.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-03-28T18:25:30.473 INFO:teuthology.orchestra.run.smithi192.stdout:(75/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 177 kB/s | 41 kB 00:00 2024-03-28T18:25:30.497 INFO:teuthology.orchestra.run.smithi192.stdout:(76/174): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 39 kB 00:00 2024-03-28T18:25:30.665 INFO:teuthology.orchestra.run.smithi192.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-28T18:25:30.690 INFO:teuthology.orchestra.run.smithi192.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-28T18:25:30.707 INFO:teuthology.orchestra.run.smithi192.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 272 kB/s | 56 kB 00:00 2024-03-28T18:25:30.732 INFO:teuthology.orchestra.run.smithi192.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-28T18:25:30.757 INFO:teuthology.orchestra.run.smithi192.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-03-28T18:25:30.782 INFO:teuthology.orchestra.run.smithi192.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T18:25:30.807 INFO:teuthology.orchestra.run.smithi192.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T18:25:30.841 INFO:teuthology.orchestra.run.smithi192.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T18:25:30.867 INFO:teuthology.orchestra.run.smithi192.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-28T18:25:30.883 INFO:teuthology.orchestra.run.smithi192.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-28T18:25:30.907 INFO:teuthology.orchestra.run.smithi145.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 940 kB/s | 548 kB 00:00 2024-03-28T18:25:30.909 INFO:teuthology.orchestra.run.smithi192.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-28T18:25:31.017 INFO:teuthology.orchestra.run.smithi192.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-28T18:25:31.041 INFO:teuthology.orchestra.run.smithi145.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 529 kB/s | 309 kB 00:00 2024-03-28T18:25:31.043 INFO:teuthology.orchestra.run.smithi192.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-28T18:25:31.093 INFO:teuthology.orchestra.run.smithi192.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T18:25:31.127 INFO:teuthology.orchestra.run.smithi192.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-03-28T18:25:31.152 INFO:teuthology.orchestra.run.smithi192.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-03-28T18:25:31.175 INFO:teuthology.orchestra.run.smithi145.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-28T18:25:31.186 INFO:teuthology.orchestra.run.smithi192.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-28T18:25:31.319 INFO:teuthology.orchestra.run.smithi192.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-03-28T18:25:31.378 INFO:teuthology.orchestra.run.smithi192.stdout:(95/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-28T18:25:31.403 INFO:teuthology.orchestra.run.smithi192.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-03-28T18:25:31.442 INFO:teuthology.orchestra.run.smithi145.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-28T18:25:31.445 INFO:teuthology.orchestra.run.smithi192.stdout:(97/174): postgresql-13.14-1.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-03-28T18:25:31.471 INFO:teuthology.orchestra.run.smithi192.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-28T18:25:31.504 INFO:teuthology.orchestra.run.smithi192.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-03-28T18:25:31.530 INFO:teuthology.orchestra.run.smithi192.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-03-28T18:25:31.555 INFO:teuthology.orchestra.run.smithi192.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 717 kB/s | 36 kB 00:00 2024-03-28T18:25:31.580 INFO:teuthology.orchestra.run.smithi192.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-03-28T18:25:31.605 INFO:teuthology.orchestra.run.smithi192.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T18:25:31.697 INFO:teuthology.orchestra.run.smithi192.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 18 MB/s | 5.8 MB 00:00 2024-03-28T18:25:31.756 INFO:teuthology.orchestra.run.smithi192.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-28T18:25:31.792 INFO:teuthology.orchestra.run.smithi145.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-28T18:25:31.956 INFO:teuthology.orchestra.run.smithi192.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-28T18:25:32.134 INFO:teuthology.orchestra.run.smithi145.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-28T18:25:32.217 INFO:teuthology.orchestra.run.smithi145.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-28T18:25:32.281 INFO:teuthology.orchestra.run.smithi192.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 60 kB/s | 19 kB 00:00 2024-03-28T18:25:32.373 INFO:teuthology.orchestra.run.smithi192.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 501 kB/s | 309 kB 00:00 2024-03-28T18:25:32.474 INFO:teuthology.orchestra.run.smithi192.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 707 kB/s | 548 kB 00:00 2024-03-28T18:25:32.499 INFO:teuthology.orchestra.run.smithi192.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-03-28T18:25:32.607 INFO:teuthology.orchestra.run.smithi192.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 363 kB/s | 49 kB 00:00 2024-03-28T18:25:32.609 INFO:teuthology.orchestra.run.smithi145.stdout:(113/174): postgresql-server-13.14-1.el9.x86_64 2.0 MB/s | 5.8 MB 00:02 2024-03-28T18:25:32.633 INFO:teuthology.orchestra.run.smithi192.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-28T18:25:32.634 INFO:teuthology.orchestra.run.smithi145.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 302 kB/s | 151 kB 00:00 2024-03-28T18:25:32.750 INFO:teuthology.orchestra.run.smithi192.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T18:25:32.885 INFO:teuthology.orchestra.run.smithi145.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-28T18:25:32.975 INFO:teuthology.orchestra.run.smithi192.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-28T18:25:32.985 INFO:teuthology.orchestra.run.smithi145.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 124 kB/s | 43 kB 00:00 2024-03-28T18:25:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-03-28T18:25:33.093 INFO:teuthology.orchestra.run.smithi192.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 372 kB/s | 43 kB 00:00 2024-03-28T18:25:33.130 INFO:teuthology.orchestra.run.smithi192.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 249 kB/s | 32 kB 00:00 2024-03-28T18:25:33.169 INFO:teuthology.orchestra.run.smithi145.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 113 kB/s | 32 kB 00:00 2024-03-28T18:25:33.230 INFO:teuthology.orchestra.run.smithi192.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 109 kB/s | 15 kB 00:00 2024-03-28T18:25:33.260 INFO:teuthology.orchestra.run.smithi145.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 54 kB/s | 15 kB 00:00 2024-03-28T18:25:33.286 INFO:teuthology.orchestra.run.smithi145.stdout:(119/174): python3-asyncssh-2.13.2-5.el9.noarch 811 kB/s | 548 kB 00:00 2024-03-28T18:25:33.286 INFO:teuthology.orchestra.run.smithi192.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-28T18:25:33.373 INFO:teuthology.orchestra.run.smithi192.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-28T18:25:33.414 INFO:teuthology.orchestra.run.smithi192.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.3 MB/s | 303 kB 00:00 2024-03-28T18:25:33.490 INFO:teuthology.orchestra.run.smithi192.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-28T18:25:33.531 INFO:teuthology.orchestra.run.smithi192.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-28T18:25:33.600 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-28T18:25:33.611 INFO:teuthology.orchestra.run.smithi145.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 389 kB/s | 172 kB 00:00 2024-03-28T18:25:33.673 INFO:teuthology.orchestra.run.smithi192.stdout:(124/174): libarrow-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-03-28T18:25:33.699 INFO:teuthology.orchestra.run.smithi192.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 867 kB/s | 144 kB 00:00 2024-03-28T18:25:33.706 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:25:33.732 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:25:33.763 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:33.772 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:33.778 INFO:teuthology.orchestra.run.smithi145.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 693 kB/s | 358 kB 00:00 2024-03-28T18:25:33.799 INFO:teuthology.orchestra.run.smithi192.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-03-28T18:25:33.803 INFO:teuthology.orchestra.run.smithi145.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 586 kB/s | 303 kB 00:00 2024-03-28T18:25:33.821 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:25:33.824 INFO:teuthology.orchestra.run.smithi192.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-03-28T18:25:33.874 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:25:33.878 INFO:teuthology.orchestra.run.smithi145.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 132 kB/s | 35 kB 00:00 2024-03-28T18:25:33.908 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:25:33.916 INFO:teuthology.orchestra.run.smithi192.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-28T18:25:33.941 INFO:teuthology.orchestra.run.smithi192.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-28T18:25:33.942 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:25:33.985 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:25:34.025 INFO:teuthology.orchestra.run.smithi192.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-03-28T18:25:34.050 INFO:teuthology.orchestra.run.smithi192.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-03-28T18:25:34.142 INFO:teuthology.orchestra.run.smithi192.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-03-28T18:25:34.184 INFO:teuthology.orchestra.run.smithi192.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 590 kB/s | 79 kB 00:00 2024-03-28T18:25:34.268 INFO:teuthology.orchestra.run.smithi192.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 461 kB/s | 58 kB 00:00 2024-03-28T18:25:34.301 INFO:teuthology.orchestra.run.smithi192.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-03-28T18:25:34.305 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:25:34.312 INFO:teuthology.orchestra.run.smithi145.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 380 kB/s | 203 kB 00:00 2024-03-28T18:25:34.327 INFO:teuthology.orchestra.run.smithi192.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-28T18:25:34.337 INFO:teuthology.orchestra.run.smithi145.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 315 kB/s | 144 kB 00:00 2024-03-28T18:25:34.369 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:34.385 INFO:teuthology.orchestra.run.smithi192.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-03-28T18:25:34.419 INFO:teuthology.orchestra.run.smithi192.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-03-28T18:25:34.444 INFO:teuthology.orchestra.run.smithi192.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-28T18:25:34.490 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:34.494 INFO:teuthology.orchestra.run.smithi192.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-03-28T18:25:34.505 INFO:teuthology.orchestra.run.smithi145.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 56 kB/s | 11 kB 00:00 2024-03-28T18:25:34.528 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:34.536 INFO:teuthology.orchestra.run.smithi192.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-28T18:25:34.553 INFO:teuthology.orchestra.run.smithi192.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 542 kB/s | 59 kB 00:00 2024-03-28T18:25:34.604 INFO:teuthology.orchestra.run.smithi192.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-03-28T18:25:34.605 INFO:teuthology.orchestra.run.smithi145.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 66 kB/s | 18 kB 00:00 2024-03-28T18:25:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:34.646 INFO:teuthology.orchestra.run.smithi192.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-03-28T18:25:34.647 INFO:teuthology.orchestra.run.smithi145.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.4 MB/s | 2.0 MB 00:00 2024-03-28T18:25:34.657 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:25:34.671 INFO:teuthology.orchestra.run.smithi192.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-28T18:25:34.697 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:25:34.721 INFO:teuthology.orchestra.run.smithi192.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 766 kB/s | 90 kB 00:00 2024-03-28T18:25:34.771 INFO:teuthology.orchestra.run.smithi192.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-28T18:25:34.797 INFO:teuthology.orchestra.run.smithi192.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 159 kB/s | 20 kB 00:00 2024-03-28T18:25:34.797 INFO:teuthology.orchestra.run.smithi145.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 79 kB/s | 23 kB 00:00 2024-03-28T18:25:34.839 INFO:teuthology.orchestra.run.smithi192.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-28T18:25:34.872 INFO:teuthology.orchestra.run.smithi145.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-03-28T18:25:34.886 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:25:34.914 INFO:teuthology.orchestra.run.smithi145.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-03-28T18:25:34.946 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:25:34.975 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:35.070 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:35.126 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:35.172 INFO:teuthology.orchestra.run.smithi192.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-03-28T18:25:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:35.213 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:25:35.214 INFO:teuthology.orchestra.run.smithi145.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 135 kB/s | 46 kB 00:00 2024-03-28T18:25:35.236 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:35.252 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:35.256 INFO:teuthology.orchestra.run.smithi192.stdout:(151/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.1 MB/s | 3.1 MB 00:00 2024-03-28T18:25:35.256 INFO:teuthology.orchestra.run.smithi145.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 230 kB/s | 79 kB 00:00 2024-03-28T18:25:35.331 INFO:teuthology.orchestra.run.smithi192.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.0 MB/s | 321 kB 00:00 2024-03-28T18:25:35.444 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:35.473 INFO:teuthology.orchestra.run.smithi192.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 5.0 MB/s | 3.4 MB 00:00 2024-03-28T18:25:35.475 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:35.490 INFO:teuthology.orchestra.run.smithi145.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-28T18:25:35.507 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:25:35.523 INFO:teuthology.orchestra.run.smithi192.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 8.7 MB/s | 2.3 MB 00:00 2024-03-28T18:25:35.557 INFO:teuthology.orchestra.run.smithi145.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 169 kB/s | 58 kB 00:00 2024-03-28T18:25:35.612 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:25:35.665 INFO:teuthology.orchestra.run.smithi192.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 219 kB/s | 42 kB 00:00 2024-03-28T18:25:35.690 INFO:teuthology.orchestra.run.smithi192.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-28T18:25:35.719 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:35.732 INFO:teuthology.orchestra.run.smithi192.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.6 MB/s | 1.8 MB 00:00 2024-03-28T18:25:35.757 INFO:teuthology.orchestra.run.smithi145.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-03-28T18:25:35.757 INFO:teuthology.orchestra.run.smithi192.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 545 kB/s | 36 kB 00:00 2024-03-28T18:25:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:35.782 INFO:teuthology.orchestra.run.smithi145.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 518 kB/s | 272 kB 00:00 2024-03-28T18:25:35.783 INFO:teuthology.orchestra.run.smithi192.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 608 kB/s | 71 kB 00:00 2024-03-28T18:25:35.808 INFO:teuthology.orchestra.run.smithi192.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-28T18:25:35.833 INFO:teuthology.orchestra.run.smithi192.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-03-28T18:25:35.849 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:35.858 INFO:teuthology.orchestra.run.smithi192.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-03-28T18:25:35.900 INFO:teuthology.orchestra.run.smithi192.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-28T18:25:35.904 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:35.908 INFO:teuthology.orchestra.run.smithi145.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 256 kB/s | 90 kB 00:00 2024-03-28T18:25:35.926 INFO:teuthology.orchestra.run.smithi192.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-28T18:25:35.951 INFO:teuthology.orchestra.run.smithi192.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-28T18:25:35.976 INFO:teuthology.orchestra.run.smithi192.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 575 kB/s | 29 kB 00:00 2024-03-28T18:25:36.001 INFO:teuthology.orchestra.run.smithi192.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 635 kB/s | 64 kB 00:00 2024-03-28T18:25:36.025 INFO:teuthology.orchestra.run.smithi145.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-03-28T18:25:36.050 INFO:teuthology.orchestra.run.smithi145.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-03-28T18:25:36.076 INFO:teuthology.orchestra.run.smithi192.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-28T18:25:36.117 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:36.160 INFO:teuthology.orchestra.run.smithi192.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-28T18:25:36.185 INFO:teuthology.orchestra.run.smithi192.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-28T18:25:36.219 INFO:teuthology.orchestra.run.smithi192.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 532 kB/s | 31 kB 00:00 2024-03-28T18:25:36.230 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:25:36.261 INFO:teuthology.orchestra.run.smithi192.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.8 MB/s | 2.3 MB 00:00 2024-03-28T18:25:36.317 INFO:teuthology.orchestra.run.smithi145.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-03-28T18:25:36.332 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:25:36.336 INFO:teuthology.orchestra.run.smithi192.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 634 kB/s | 74 kB 00:00 2024-03-28T18:25:36.342 INFO:teuthology.orchestra.run.smithi145.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 434 kB/s | 188 kB 00:00 2024-03-28T18:25:36.367 INFO:teuthology.orchestra.run.smithi145.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-28T18:25:36.368 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:25:36.395 INFO:teuthology.orchestra.run.smithi192.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-03-28T18:25:36.399 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:25:36.399 INFO:teuthology.orchestra.run.smithi192.stdout:Total 12 MB/s | 225 MB 00:18 2024-03-28T18:25:36.411 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:25:36.455 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:25:36.530 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:25:36.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:25:36.639 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:25:36.667 INFO:teuthology.orchestra.run.smithi145.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 145 kB/s | 51 kB 00:00 2024-03-28T18:25:36.703 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:25:36.709 INFO:teuthology.orchestra.run.smithi145.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 262 kB/s | 90 kB 00:00 2024-03-28T18:25:36.738 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:25:36.781 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:25:36.814 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:25:36.842 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:25:36.860 INFO:teuthology.orchestra.run.smithi145.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 444 kB/s | 230 kB 00:00 2024-03-28T18:25:36.911 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:25:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:25:36.968 INFO:teuthology.orchestra.run.smithi145.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 77 kB/s | 20 kB 00:00 2024-03-28T18:25:36.986 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-28T18:25:37.007 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:25:37.044 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:25:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:25:37.158 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:25:37.232 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:25:37.252 INFO:teuthology.orchestra.run.smithi145.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 731 kB/s | 427 kB 00:00 2024-03-28T18:25:37.257 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-28T18:25:37.257 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-28T18:25:37.285 INFO:teuthology.orchestra.run.smithi145.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 450 kB/s | 192 kB 00:00 2024-03-28T18:25:37.368 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:25:37.428 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:25:37.669 INFO:teuthology.orchestra.run.smithi145.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-03-28T18:25:37.719 INFO:teuthology.orchestra.run.smithi145.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-28T18:25:37.950 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:25:38.031 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:25:38.070 INFO:teuthology.orchestra.run.smithi145.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 804 kB/s | 321 kB 00:00 2024-03-28T18:25:38.126 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:25:38.185 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:38.195 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:38.225 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:25:38.266 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:25:38.304 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:25:38.333 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:25:38.445 INFO:teuthology.orchestra.run.smithi145.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.3 MB/s | 2.3 MB 00:00 2024-03-28T18:25:38.537 INFO:teuthology.orchestra.run.smithi145.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 460 kB/s | 42 kB 00:00 2024-03-28T18:25:38.574 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:25:38.655 INFO:teuthology.orchestra.run.smithi145.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-28T18:25:38.713 INFO:teuthology.orchestra.run.smithi145.stdout:(156/174): librados2-19.0.0-2145.g3be9a5c9.el9. 2.3 MB/s | 3.4 MB 00:01 2024-03-28T18:25:38.757 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:25:38.771 INFO:teuthology.orchestra.run.smithi145.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 611 kB/s | 71 kB 00:00 2024-03-28T18:25:38.822 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:25:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:25:38.888 INFO:teuthology.orchestra.run.smithi145.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-28T18:25:38.955 INFO:teuthology.orchestra.run.smithi145.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 150 kB/s | 36 kB 00:00 2024-03-28T18:25:38.981 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-28T18:25:38.982 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-28T18:25:38.989 INFO:teuthology.orchestra.run.smithi145.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-28T18:25:38.994 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:25:39.053 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:25:39.055 INFO:teuthology.orchestra.run.smithi145.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-28T18:25:39.108 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:25:39.122 INFO:teuthology.orchestra.run.smithi145.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 811 kB/s | 108 kB 00:00 2024-03-28T18:25:39.181 INFO:teuthology.orchestra.run.smithi145.stdout:(163/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.6 MB/s | 1.8 MB 00:01 2024-03-28T18:25:39.248 INFO:teuthology.orchestra.run.smithi145.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 805 kB/s | 155 kB 00:00 2024-03-28T18:25:39.273 INFO:teuthology.orchestra.run.smithi145.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 901 kB/s | 136 kB 00:00 2024-03-28T18:25:39.299 INFO:teuthology.orchestra.run.smithi145.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 546 kB/s | 64 kB 00:00 2024-03-28T18:25:39.341 INFO:teuthology.orchestra.run.smithi145.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 311 kB/s | 29 kB 00:00 2024-03-28T18:25:39.410 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:25:39.474 INFO:teuthology.orchestra.run.smithi145.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-28T18:25:39.480 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:25:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:25:39.550 INFO:teuthology.orchestra.run.smithi145.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.1 MB/s | 465 kB 00:00 2024-03-28T18:25:39.563 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:25:39.583 INFO:teuthology.orchestra.run.smithi145.stdout:(170/174): openssl-3.0.7-27.el9.x86_64.rpm 3.9 MB/s | 1.2 MB 00:00 2024-03-28T18:25:39.598 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:25:39.625 INFO:teuthology.orchestra.run.smithi145.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-28T18:25:39.642 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:25:39.684 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:25:39.750 INFO:teuthology.orchestra.run.smithi145.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 2.2 MB 00:00 2024-03-28T18:25:39.784 INFO:teuthology.orchestra.run.smithi145.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-28T18:25:39.827 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:25:39.880 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:25:39.911 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:25:39.952 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:25:40.046 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:25:40.051 INFO:teuthology.orchestra.run.smithi145.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 175 kB/s | 74 kB 00:00 2024-03-28T18:25:40.058 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:25:40.058 INFO:teuthology.orchestra.run.smithi145.stdout:Total 9.9 MB/s | 225 MB 00:22 2024-03-28T18:25:40.086 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:25:40.126 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:25:40.152 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:25:40.176 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:25:40.265 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:25:40.384 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:25:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:25:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:25:40.583 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:25:40.618 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-28T18:25:40.639 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:25:40.830 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:25:40.881 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:25:40.899 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-28T18:25:40.900 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-28T18:25:41.009 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:25:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:25:41.086 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:25:41.134 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:25:41.179 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:25:41.213 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:25:41.252 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:25:41.292 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:25:41.330 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:25:41.406 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:25:41.441 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:25:41.474 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:25:41.516 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:25:41.550 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:25:41.639 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:25:41.720 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:25:41.764 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:25:41.804 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:25:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:25:41.949 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:25:41.997 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:25:42.724 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:25:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:25:42.813 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:25:42.869 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:25:42.955 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-28T18:25:42.956 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-28T18:25:42.965 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:25:43.005 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:25:43.181 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:25:43.211 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:25:43.288 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:25:43.931 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:25:44.049 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:25:45.269 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:25:45.355 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:25:45.385 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:25:45.416 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:25:45.443 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:25:45.756 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:25:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:25:45.855 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:25:45.887 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:25:45.893 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-28T18:25:45.923 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:25:46.016 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:25:46.061 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:25:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:46.218 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:46.386 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:25:46.405 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:25:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:25:46.459 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:25:46.489 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:25:46.492 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:25:46.518 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:25:46.520 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:25:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:25:46.885 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:25:46.952 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:47.106 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:47.195 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:47.224 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:25:47.258 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:25:47.424 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:25:47.480 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:25:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:47.611 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:47.634 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:47.698 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:47.738 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:25:47.761 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:47.777 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:47.950 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:47.998 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:25:48.201 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:48.261 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:48.333 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:48.387 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:48.604 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:48.691 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:25:48.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:25:48.732 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:25:48.837 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:25:48.875 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:25:48.902 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:25:48.937 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:25:49.012 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:25:49.065 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:25:49.104 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:25:49.167 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:25:49.202 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:25:49.246 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:25:49.261 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:25:49.279 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:25:49.295 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:25:49.307 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:25:49.375 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:25:49.376 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:25:49.377 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:25:49.377 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:25:49.377 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:25:49.405 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:25:49.412 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:25:49.455 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:25:49.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:25:49.552 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:25:49.598 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:25:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:25:49.802 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:25:49.850 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-28T18:25:50.213 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:25:50.249 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:25:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:25:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:25:50.366 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:25:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:25:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:25:50.445 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:25:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:25:50.531 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:25:50.534 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:25:50.582 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:50.592 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:50.630 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:25:50.680 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:25:50.717 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:25:50.747 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:25:50.863 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:25:50.917 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:25:51.022 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:25:51.052 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:51.143 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:25:51.163 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:25:51.176 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:25:51.216 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:25:51.223 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:25:51.261 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:25:51.389 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:25:51.415 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:25:51.430 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:25:51.463 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:25:51.482 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:51.484 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:25:51.566 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:25:51.569 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:25:51.591 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:51.596 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:25:51.633 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:25:51.640 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:25:51.679 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:25:51.680 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:25:51.702 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:51.717 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:25:51.740 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:25:51.783 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:25:51.832 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:25:51.860 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:51.899 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:25:51.939 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:25:51.968 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:25:52.023 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:25:52.054 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:25:52.096 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:25:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:25:52.190 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:25:52.228 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:25:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:52.260 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:25:52.286 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:25:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:25:52.310 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:25:52.382 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:25:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:52.511 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:25:52.619 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:25:52.661 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:25:52.673 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:25:52.725 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:25:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:25:52.780 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:25:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:25:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:25:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:25:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:25:52.963 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:25:53.006 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:25:53.031 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:25:53.085 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:25:53.123 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:25:53.134 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:25:53.170 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:25:53.188 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:25:53.203 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:25:53.222 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:25:53.242 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:25:53.266 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:25:53.270 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:25:53.299 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:25:53.304 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:25:53.335 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:25:53.343 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:25:53.375 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:25:53.405 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:25:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:25:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:25:53.481 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:25:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:25:53.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:25:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:25:53.548 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:25:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:25:53.590 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:25:53.624 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:25:53.636 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:25:53.709 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:25:53.715 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:25:53.803 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:25:53.840 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:25:53.847 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:25:53.879 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:25:53.895 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:25:53.926 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:25:54.023 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:25:54.071 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:25:54.414 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:25:54.491 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:25:54.599 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:25:54.636 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:54.646 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:25:54.693 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:25:54.734 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:25:54.772 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:25:54.781 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:25:54.802 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:25:54.827 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:25:54.860 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:25:54.909 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:25:54.990 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:25:55.020 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:25:55.080 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:25:55.197 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:25:55.217 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:25:55.252 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:25:55.295 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:25:55.338 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:25:55.381 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:25:55.540 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:25:55.589 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:25:55.640 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:25:55.713 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:25:55.745 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:25:55.782 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:25:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:25:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:25:55.907 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:25:55.933 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:25:55.949 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:25:56.048 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:25:56.081 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:25:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:25:56.160 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:25:56.201 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:25:56.295 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:25:56.335 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:25:56.375 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:25:56.401 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:25:56.425 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:25:56.502 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:25:56.626 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:25:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:25:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:25:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:25:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:25:57.071 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:25:57.139 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:25:57.275 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:25:57.276 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:25:57.320 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:25:57.351 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:25:57.368 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:25:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:25:57.399 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:25:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:25:57.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:25:57.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:25:57.452 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:25:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:25:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:25:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:25:57.629 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:25:57.663 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:25:57.696 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:25:57.730 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:25:57.754 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:25:57.764 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:25:57.779 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:25:57.853 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:25:57.869 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:25:57.900 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:25:57.937 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:25:57.942 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:25:57.995 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:25:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:25:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:25:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:25:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:25:58.399 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:25:58.424 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:25:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:25:58.517 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:25:58.947 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:25:58.992 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:25:59.026 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:25:59.073 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:25:59.162 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:25:59.210 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:25:59.387 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:25:59.416 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:25:59.493 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:26:00.141 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:26:00.245 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:26:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:01.046 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-03-28T18:26:01.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:01.211 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:01.227 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:01.227 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:01.227 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:26:01.228 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:26:01.364 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:26:01.364 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:26:01.364 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:26:01.364 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:01.392 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:26:01.446 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:26:01.524 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:26:01.555 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:26:01.586 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:26:01.614 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:26:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:26:01.961 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:26:02.051 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:26:02.082 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:26:02.118 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:26:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:26:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:26:02.668 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:26:02.690 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:26:02.715 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:02.741 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:02.957 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:02.980 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:03.424 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:03.447 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:26:03.945 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:03.977 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:04.009 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:26:04.068 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:26:04.091 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:04.326 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:04.359 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:04.862 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:05.106 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:05.323 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:05.351 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:05.451 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:05.614 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:26:05.645 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:05.651 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:26:05.668 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:05.669 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:05.669 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:26:05.669 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:05.669 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:05.669 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:05.689 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:26:05.712 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:05.723 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:26:05.785 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:26:05.785 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:26:05.785 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:26:05.786 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:05.824 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:26:06.655 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:26:06.681 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:06.704 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:06.749 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:06.770 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:26:06.771 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:06.847 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:07.150 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:07.173 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:11.662 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:26:11.693 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:26:11.720 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:26:11.901 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:26:12.020 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:26:12.060 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:26:12.099 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:26:12.152 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:26:12.195 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:26:12.228 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:26:12.268 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:26:12.298 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:26:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:26:12.422 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:26:12.472 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:26:13.066 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:26:13.066 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-03-28T18:26:13.066 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-03-28T18:26:13.066 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-03-28T18:26:13.067 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-03-28T18:26:13.067 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-03-28T18:26:13.067 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-03-28T18:26:13.067 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-03-28T18:26:13.067 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:13.091 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:13.129 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:13.148 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:13.232 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:13.248 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:13.248 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:26:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:13.973 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:26:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:26:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:26:14.195 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:26:14.236 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:26:14.294 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:26:14.320 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:26:14.346 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:26:14.346 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:14.443 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:14.473 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:26:14.473 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:14.540 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:14.575 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:26:14.617 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:26:14.642 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:26:14.667 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:26:14.700 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:26:14.748 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:26:14.754 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:14.780 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:14.789 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:26:14.814 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:26:14.839 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:26:14.865 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:14.890 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:26:14.896 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:14.897 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:15.076 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:15.076 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:15.159 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:15.184 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:15.216 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:26:15.243 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:26:15.269 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:26:15.332 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:15.461 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:15.484 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:15.484 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:15.484 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:26:15.484 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:15.484 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:15.485 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:15.571 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:26:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:26:16.011 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:26:16.034 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:16.269 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:16.302 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:17.260 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:17.286 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:17.386 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:17.548 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:26:17.571 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:17.594 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:17.638 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:17.650 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:17.686 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:26:17.825 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:26:17.825 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2024-03-28T18:26:17.825 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /root 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2024-03-28T18:26:17.826 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:18.002 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:26:18.002 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:18.002 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:26:18.002 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:18.574 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:26:18.616 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:18.639 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:18.705 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:18.705 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:26:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:18.750 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:18.772 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:26:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:18.773 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:19.498 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:19.523 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:26:19.523 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:19.524 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:26:19.524 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:19.524 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:26:19.524 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:19.748 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:26:19.771 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:20.207 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:20.229 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:26:20.229 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:20.229 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:26:20.229 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:20.230 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:26:20.230 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:20.325 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:26:20.447 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:26:20.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:26:20.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:26:20.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:26:20.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:26:20.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:26:20.453 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:26:20.454 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:26:20.455 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:26:20.456 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:26:20.457 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:26:20.458 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:26:20.459 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:26:20.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:26:20.461 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:26:20.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:26:20.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:26:20.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:26:20.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:20.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:26:20.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:26:20.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:26:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:26:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:26:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:26:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:21.155 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:26:21.189 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:26:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:22.281 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:26:22.434 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:26:22.466 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:26:22.489 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:22.489 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:26:22.489 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:22.489 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:26:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:26:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:26:23.288 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:23.289 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.290 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.291 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:26:23.292 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.293 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:26:23.294 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:26:23.295 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:26:23.296 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:26:23.297 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:26:23.298 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.299 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.300 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:26:23.301 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-28T18:26:23.302 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-28T18:26:23.475 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:26:23.511 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:26:23.534 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:23.547 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:23.578 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:23.597 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:26:23.600 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:26:23.600 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:23.600 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:26:23.601 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:23.627 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:26:23.646 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:23.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:26:23.668 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:23.668 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:23.668 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:26:23.668 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:23.668 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:26:23.669 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:23.773 DEBUG:teuthology.parallel:result is None 2024-03-28T18:26:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:26:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:26:23.972 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:23.994 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:26:23.994 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:26:23.994 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:26:23.995 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:23.995 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:26:23.995 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:26:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:26:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:26:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:26:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:26:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:26:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:26:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:26:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:26:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:26:24.959 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:25.827 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:26:25.914 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:26:25.954 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:26:26.035 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:26:26.068 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:26:26.092 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:26:26.118 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:26:26.144 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:26:26.145 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:26.216 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:26.245 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:26:26.245 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:26.322 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:26.347 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:26:26.381 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:26:26.406 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:26:26.431 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:26:26.463 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:26:26.503 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:26:26.536 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:26:26.561 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:26:26.586 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:26:26.612 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:26.637 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:26:26.644 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:26.644 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:26.914 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:26.939 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:26.962 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:26:26.990 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:26:27.015 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:26:27.079 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:28.450 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:28.500 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:26:28.539 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:26:28.566 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:26:28.746 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:26:28.866 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:26:28.905 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:26:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:26:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:26:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:26:29.098 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:26:29.130 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:26:29.160 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:26:29.230 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:26:29.289 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:26:29.325 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:26:29.930 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:29.973 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:29.993 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:26:30.803 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:26:30.893 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:26:30.934 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:26:31.024 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:26:31.055 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:26:31.081 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:26:31.105 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:26:31.132 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:26:31.132 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:31.219 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:26:31.258 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:26:31.258 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:31.351 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:26:31.377 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:26:31.435 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:26:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:26:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:26:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:26:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:26:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:26:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:26:31.659 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:26:31.685 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:31.710 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:26:31.717 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:31.717 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:31.904 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:26:31.904 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:31.987 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:26:32.012 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:32.045 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:26:32.071 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:26:32.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:26:32.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:26:32.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:26:32.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:26:32.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:26:32.099 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:26:32.109 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:26:32.109 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:26:32.109 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:26:32.109 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:26:32.110 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:26:32.111 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:26:32.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:26:32.113 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:26:32.114 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:26:32.115 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:26:32.116 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:26:32.117 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:26:32.118 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:26:32.119 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:26:32.120 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:26:32.121 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:26:32.163 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:35.075 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:26:35.076 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.077 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.079 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:26:35.080 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.081 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:26:35.082 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:26:35.083 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.084 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:26:35.085 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:26:35.086 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:26:35.087 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:26:35.088 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.089 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:26:35.090 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:26:35.091 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-28T18:26:35.575 DEBUG:teuthology.parallel:result is None 2024-03-28T18:26:37.166 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:26:37.166 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:26:37.166 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:26:37.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:26:37.168 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:26:37.169 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:26:37.170 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:26:37.171 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:26:37.172 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:26:37.173 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:26:37.174 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:26:37.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:26:37.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:26:37.177 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:26:37.178 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:26:37.179 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:26:37.180 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:26:37.180 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:26:37.180 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:26:37.180 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:26:37.180 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:26:40.147 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:26:40.147 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:40.147 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2024-03-28T18:26:40.147 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:40.147 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.148 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.149 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.150 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:26:40.151 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:26:40.152 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:26:40.153 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:26:40.154 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:26:40.155 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:26:40.156 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:26:40.157 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:26:40.158 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:26:40.159 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-28T18:26:40.625 DEBUG:teuthology.parallel:result is None 2024-03-28T18:26:40.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:40.768 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:26:40.789 INFO:teuthology.orchestra.run.smithi145.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:40.790 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:40.790 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:26:40.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:40.932 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:26:40.957 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:40.957 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:40.957 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:26:40.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:41.177 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:26:41.202 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:41.202 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:26:41.203 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:26:41.204 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T18:26:41.204 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:26:41.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:26:41.234 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:26:41.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:26:41.269 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:26:41.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:26:41.303 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T18:26:41.303 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:26:41.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:26:41.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:26:41.398 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:26:41.398 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:26:41.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:26:41.494 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:26:41.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:26:41.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:26:41.595 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T18:26:41.596 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:26:41.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:26:41.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:26:41.690 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:26:41.690 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:26:41.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:26:41.784 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:26:41.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:26:41.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:26:41.885 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T18:26:41.885 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:26:41.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:26:41.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:26:41.979 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:26:41.980 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:26:42.006 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:26:42.073 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:26:42.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:26:42.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:26:42.173 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T18:26:42.261 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-28T18:26:42.261 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:42.262 INFO:tasks.cephadm:Cluster fsid is b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:26:42.262 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T18:26:42.262 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.145', 'mon.b': '172.21.15.164', 'mon.c': '172.21.15.192'} 2024-03-28T18:26:42.262 INFO:tasks.cephadm:First mon is mon.a on smithi145 2024-03-28T18:26:42.262 INFO:tasks.cephadm:First mgr is x 2024-03-28T18:26:42.262 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T18:26:42.263 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname $(hostname -s) 2024-03-28T18:26:42.291 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2024-03-28T18:26:42.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-03-28T18:26:42.347 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T18:26:42.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:42.486 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T18:26:42.633 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:26:42.634 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T18:26:42.634 DEBUG:teuthology.orchestra.run.smithi145:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:43.124 INFO:teuthology.orchestra.run.smithi145.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:26 /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:43.124 DEBUG:teuthology.orchestra.run.smithi164:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:43.733 INFO:teuthology.orchestra.run.smithi164.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:26 /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:43.733 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:44.188 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:26 /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:44.188 DEBUG:teuthology.orchestra.run.smithi145:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:44.207 DEBUG:teuthology.orchestra.run.smithi164:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:44.225 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:26:44.259 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T18:26:44.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:26:44.262 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:26:44.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:26:45.160 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:26:45.163 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:26:45.178 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:27:17.848 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-28T18:27:17.849 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout: "repo_digests": [ 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:27:32.999 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:27:33.000 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:27:41.287 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-28T18:27:41.287 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:27:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:27:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "repo_digests": [ 2024-03-28T18:27:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:27:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-03-28T18:27:41.288 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-28T18:27:41.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph 2024-03-28T18:27:41.349 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2024-03-28T18:27:41.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-03-28T18:27:41.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /etc/ceph 2024-03-28T18:27:41.444 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2024-03-28T18:27:41.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-03-28T18:27:41.501 INFO:tasks.cephadm:Writing seed config... 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T18:27:41.503 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T18:27:41.504 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T18:27:41.505 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T18:27:41.506 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:27:41.506 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T18:27:41.523 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b991dbde-ed30-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T18:27:41.523 DEBUG:teuthology.orchestra.run.smithi145:mon.a> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T18:27:41.566 DEBUG:teuthology.orchestra.run.smithi145:mgr.x> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T18:27:41.608 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T18:27:41.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.145 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:27:41.802 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:27:41.802 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'b991dbde-ed30-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.145', '--skip-admin-label'] 2024-03-28T18:27:41.826 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:27:41.827 INFO:teuthology.orchestra.run.smithi145.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T18:27:41.827 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying podman|docker is present... 2024-03-28T18:27:41.850 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:27:41.850 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying lvm2 is present... 2024-03-28T18:27:41.850 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying time synchronization is in place... 2024-03-28T18:27:41.857 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T18:27:41.858 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T18:27:41.863 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T18:27:41.863 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:27:41.870 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout enabled 2024-03-28T18:27:41.876 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout active 2024-03-28T18:27:41.876 INFO:teuthology.orchestra.run.smithi145.stdout:Unit chronyd.service is enabled and running 2024-03-28T18:27:41.876 INFO:teuthology.orchestra.run.smithi145.stdout:Repeating the final host check... 2024-03-28T18:27:41.899 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:27:41.899 INFO:teuthology.orchestra.run.smithi145.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T18:27:41.899 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl is present 2024-03-28T18:27:41.899 INFO:teuthology.orchestra.run.smithi145.stdout:lvcreate is present 2024-03-28T18:27:41.906 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T18:27:41.906 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T18:27:41.911 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T18:27:41.912 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:27:41.918 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout enabled 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout active 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:Unit chronyd.service is enabled and running 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:Host looks OK 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:Cluster fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:Acquiring lock 140024433782208 on /run/cephadm/b991dbde-ed30-11ee-95d0-87774f69a715.lock 2024-03-28T18:27:41.924 INFO:teuthology.orchestra.run.smithi145.stdout:Lock 140024433782208 acquired on /run/cephadm/b991dbde-ed30-11ee-95d0-87774f69a715.lock 2024-03-28T18:27:41.925 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying IP 172.21.15.145 port 3300 ... 2024-03-28T18:27:41.925 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying IP 172.21.15.145 port 6789 ... 2024-03-28T18:27:41.925 INFO:teuthology.orchestra.run.smithi145.stdout:Base mon IP(s) is [172.21.15.145:3300, 172.21.15.145:6789], mon addrv is [v2:172.21.15.145:3300,v1:172.21.15.145:6789] 2024-03-28T18:27:41.928 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.145 metric 100 2024-03-28T18:27:41.928 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.145 metric 100 2024-03-28T18:27:41.931 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T18:27:41.931 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-28T18:27:41.931 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-03-28T18:27:41.933 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T18:27:41.933 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T18:27:41.933 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T18:27:41.933 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-28T18:27:41.934 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-03-28T18:27:41.934 INFO:teuthology.orchestra.run.smithi145.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T18:27:41.934 INFO:teuthology.orchestra.run.smithi145.stdout:Mon IP `172.21.15.145` is in CIDR network `172.21.0.0/20` 2024-03-28T18:27:41.934 INFO:teuthology.orchestra.run.smithi145.stdout:Mon IP `172.21.15.145` is in CIDR network `172.21.0.0/20` 2024-03-28T18:27:41.934 INFO:teuthology.orchestra.run.smithi145.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T18:27:41.935 INFO:teuthology.orchestra.run.smithi145.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T18:27:41.936 INFO:teuthology.orchestra.run.smithi145.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:27:46.286 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T18:27:46.287 INFO:teuthology.orchestra.run.smithi145.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T18:27:52.840 INFO:teuthology.orchestra.run.smithi145.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:27:52.840 INFO:teuthology.orchestra.run.smithi145.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:27:52.840 INFO:teuthology.orchestra.run.smithi145.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T18:27:53.930 INFO:teuthology.orchestra.run.smithi145.stdout:stat: stdout 167 167 2024-03-28T18:27:53.930 INFO:teuthology.orchestra.run.smithi145.stdout:Creating initial keys... 2024-03-28T18:27:54.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph-authtool: stdout AQCqtgVmQhayDBAA6LbTYyUE1IgEQHHT6Nd/cQ== 2024-03-28T18:27:55.648 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph-authtool: stdout AQCrtgVmvGGVBBAAzgQqFdVolIBzxxsLHEEZwA== 2024-03-28T18:27:56.579 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph-authtool: stdout AQCrtgVmxG+fOhAAj+G2zpa6CgWqsjuNHdM/7A== 2024-03-28T18:27:56.580 INFO:teuthology.orchestra.run.smithi145.stdout:Creating initial monmap... 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:monmaptool for a [v2:172.21.15.145:3300,v1:172.21.15.145:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:setting min_mon_release = quincy 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: set fsid to b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:27:58.834 INFO:teuthology.orchestra.run.smithi145.stdout:Creating mon... 2024-03-28T18:27:59.739 INFO:teuthology.orchestra.run.smithi145.stdout:create mon.a on 2024-03-28T18:27:59.942 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T18:28:00.099 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T18:28:00.283 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target. 2024-03-28T18:28:00.283 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target. 2024-03-28T18:28:00.519 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a 2024-03-28T18:28:00.519 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T18:28:00.680 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target.wants/ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715@.service. 2024-03-28T18:28:01.204 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:28:01.204 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout disabled 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:firewalld.service is not enabled 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mon to start... 2024-03-28T18:28:01.211 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mon... 2024-03-28T18:28:01.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:01 smithi145 systemd[1]: Started Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: monmap e1: 1 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: fsmap 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: mgrmap e1: no daemons active 2024-03-28T18:28:02.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:02 smithi145 ceph-mon[31777]: from='client.? 172.21.15.145:0/1711091827' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T18:28:02.308 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout id: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout services: 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.542682s) 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout data: 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T18:28:02.314 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.644+0000 7faa5c833700 1 Processor -- start 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.644+0000 7faa5c833700 1 -- start start 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5c833700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5c833700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa58108990 con 0x7faa58107fa0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44398/0 (socket says 172.21.15.145:44398) 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5659c700 1 -- 172.21.15.145:0/2960556156 learned_addr learned my addr 172.21.15.145:0/2960556156 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5659c700 1 -- 172.21.15.145:0/2960556156 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa58106c00 con 0x7faa58107fa0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.645+0000 7faa5659c700 1 --2- 172.21.15.145:0/2960556156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0098a0 tx=0x7faa4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be46c513b7510426 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5559a700 1 -- 172.21.15.145:0/2960556156 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa4c004160 con 0x7faa58107fa0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5559a700 1 -- 172.21.15.145:0/2960556156 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c004300 con 0x7faa58107fa0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5559a700 1 -- 172.21.15.145:0/2960556156 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa4c0036a0 con 0x7faa58107fa0 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 -- 172.21.15.145:0/2960556156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 msgr2=0x7faa581083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:02.315 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 --2- 172.21.15.145:0/2960556156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0098a0 tx=0x7faa4c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 -- 172.21.15.145:0/2960556156 shutdown_connections 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 --2- 172.21.15.145:0/2960556156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa581083c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 -- 172.21.15.145:0/2960556156 >> 172.21.15.145:0/2960556156 conn(0x7faa581016f0 msgr2=0x7faa58103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 -- 172.21.15.145:0/2960556156 shutdown_connections 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.646+0000 7faa5c833700 1 -- 172.21.15.145:0/2960556156 wait complete. 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5c833700 1 Processor -- start 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5c833700 1 -- start start 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5c833700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5c833700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa58107070 con 0x7faa58107fa0 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44412/0 (socket says 172.21.15.145:44412) 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.647+0000 7faa5659c700 1 -- 172.21.15.145:0/1711091827 learned_addr learned my addr 172.21.15.145:0/1711091827 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa5659c700 1 -- 172.21.15.145:0/1711091827 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5819b740 con 0x7faa58107fa0 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa5659c700 1 --2- 172.21.15.145:0/1711091827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faa4c000c00 tx=0x7faa4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa4c003db0 con 0x7faa58107fa0 2024-03-28T18:28:02.316 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c011630 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa4c011950 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5819b9d0 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5819e6c0 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7faa4c01a460 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.648+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faa4c01b780 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.649+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa58069250 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.651+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7faa4c031020 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.721+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7faa5802dc80 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.721+0000 7faa477fe700 1 -- 172.21.15.145:0/1711091827 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7faa4c01b9d0 con 0x7faa58107fa0 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.721+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 msgr2=0x7faa5819b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.721+0000 7faa5c833700 1 --2- 172.21.15.145:0/1711091827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faa4c000c00 tx=0x7faa4c004750 comp rx=0 tx=0).stop 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.722+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 shutdown_connections 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.722+0000 7faa5c833700 1 --2- 172.21.15.145:0/1711091827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa58107fa0 0x7faa5819b200 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.722+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 >> 172.21.15.145:0/1711091827 conn(0x7faa581016f0 msgr2=0x7faa58191d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.722+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 shutdown_connections 2024-03-28T18:28:02.318 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:01.722+0000 7faa5c833700 1 -- 172.21.15.145:0/1711091827 wait complete. 2024-03-28T18:28:02.318 INFO:teuthology.orchestra.run.smithi145.stdout:mon is available 2024-03-28T18:28:02.318 INFO:teuthology.orchestra.run.smithi145.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T18:28:03.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:03 smithi145 ceph-mon[31777]: from='client.? 172.21.15.145:0/1252303099' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T18:28:03.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:03 smithi145 ceph-mon[31777]: from='client.? 172.21.15.145:0/1252303099' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T18:28:03.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:03.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T18:28:03.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout fsid = b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:03.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.145:3300,v1:172.21.15.145:6789] 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T18:28:03.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.730+0000 7f30432be700 1 Processor -- start 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f30432be700 1 -- start start 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f30432be700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f30432be700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c07bed0 con 0x7f303c1084d0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f304105a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f304105a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39342/0 (socket says 172.21.15.145:39342) 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f304105a700 1 -- 172.21.15.145:0/2458647145 learned_addr learned my addr 172.21.15.145:0/2458647145 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f304105a700 1 -- 172.21.15.145:0/2458647145 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c10ae00 con 0x7f303c1084d0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.731+0000 7f304105a700 1 --2- 172.21.15.145:0/2458647145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30300098a0 tx=0x7f3030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9c7ceca32bd0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f302ffff700 1 -- 172.21.15.145:0/2458647145 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3030004160 con 0x7f303c1084d0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f302ffff700 1 -- 172.21.15.145:0/2458647145 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030004300 con 0x7f303c1084d0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f302ffff700 1 -- 172.21.15.145:0/2458647145 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f30300036a0 con 0x7f303c1084d0 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 -- 172.21.15.145:0/2458647145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 msgr2=0x7f303c10a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 --2- 172.21.15.145:0/2458647145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30300098a0 tx=0x7f3030009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:03.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 -- 172.21.15.145:0/2458647145 shutdown_connections 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 --2- 172.21.15.145:0/2458647145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c10a8c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 -- 172.21.15.145:0/2458647145 >> 172.21.15.145:0/2458647145 conn(0x7f303c101f20 msgr2=0x7f303c104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 -- 172.21.15.145:0/2458647145 shutdown_connections 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.732+0000 7f30432be700 1 -- 172.21.15.145:0/2458647145 wait complete. 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f30432be700 1 Processor -- start 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f30432be700 1 -- start start 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f30432be700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f30432be700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c10b2d0 con 0x7f303c1084d0 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f304105a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f304105a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39344/0 (socket says 172.21.15.145:39344) 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f304105a700 1 -- 172.21.15.145:0/1252303099 learned_addr learned my addr 172.21.15.145:0/1252303099 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.733+0000 7f304105a700 1 -- 172.21.15.145:0/1252303099 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c19b720 con 0x7f303c1084d0 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f304105a700 1 --2- 172.21.15.145:0/1252303099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3030000c00 tx=0x7f3030004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:03.541 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3030003db0 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030011630 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c19b9b0 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3030011950 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c19e650 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f303001a460 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.734+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f303001b780 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.735+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c069250 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.737+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f303001ba20 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.806+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f303c02dc80 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.809+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f303001bc00 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.809+0000 7f302e7fc700 1 -- 172.21.15.145:0/1252303099 <== mon.0 v2:172.21.15.145:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f303001bde0 con 0x7f303c1084d0 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 msgr2=0x7f303c19b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 --2- 172.21.15.145:0/1252303099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3030000c00 tx=0x7f3030004750 comp rx=0 tx=0).stop 2024-03-28T18:28:03.542 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 shutdown_connections 2024-03-28T18:28:03.543 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 --2- 172.21.15.145:0/1252303099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f303c1084d0 0x7f303c19b1e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:03.543 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 >> 172.21.15.145:0/1252303099 conn(0x7f303c101f20 msgr2=0x7f303c102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:03.543 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 shutdown_connections 2024-03-28T18:28:03.543 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:02.810+0000 7f30432be700 1 -- 172.21.15.145:0/1252303099 wait complete. 2024-03-28T18:28:03.543 INFO:teuthology.orchestra.run.smithi145.stdout:Generating new minimal ceph.conf... 2024-03-28T18:28:04.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:04 smithi145 ceph-mon[31777]: from='client.? 172.21.15.145:0/702027234' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:28:04.976 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f290ba68700 1 Processor -- start 2024-03-28T18:28:04.976 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f290ba68700 1 -- start start 2024-03-28T18:28:04.976 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f290ba68700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f290ba68700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290407bf50 con 0x7f29041085e0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f2909804700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f2909804700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39346/0 (socket says 172.21.15.145:39346) 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f2909804700 1 -- 172.21.15.145:0/1298854803 learned_addr learned my addr 172.21.15.145:0/1298854803 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.959+0000 7f2909804700 1 -- 172.21.15.145:0/1298854803 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290410af10 con 0x7f29041085e0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f2909804700 1 --2- 172.21.15.145:0/1298854803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098a0 tx=0x7f28f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8324e2e4c8c6d59d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f2908802700 1 -- 172.21.15.145:0/1298854803 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4004160 con 0x7f29041085e0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f2908802700 1 -- 172.21.15.145:0/1298854803 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28f4004300 con 0x7f29041085e0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f2908802700 1 -- 172.21.15.145:0/1298854803 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4011b50 con 0x7f29041085e0 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f290ba68700 1 -- 172.21.15.145:0/1298854803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 msgr2=0x7f290410a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f290ba68700 1 --2- 172.21.15.145:0/1298854803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098a0 tx=0x7f28f4009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f290ba68700 1 -- 172.21.15.145:0/1298854803 shutdown_connections 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f290ba68700 1 --2- 172.21.15.145:0/1298854803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290410a9d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.960+0000 7f290ba68700 1 -- 172.21.15.145:0/1298854803 >> 172.21.15.145:0/1298854803 conn(0x7f2904101f90 msgr2=0x7f29041043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:04.977 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 -- 172.21.15.145:0/1298854803 shutdown_connections 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 -- 172.21.15.145:0/1298854803 wait complete. 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 Processor -- start 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 -- start start 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.961+0000 7f290ba68700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290410bff0 con 0x7f29041085e0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f2909804700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f2909804700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39352/0 (socket says 172.21.15.145:39352) 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f2909804700 1 -- 172.21.15.145:0/702027234 learned_addr learned my addr 172.21.15.145:0/702027234 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f2909804700 1 -- 172.21.15.145:0/702027234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290419c0c0 con 0x7f29041085e0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f2909804700 1 --2- 172.21.15.145:0/702027234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fb0 tx=0x7f28f4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f40038b0 con 0x7f29041085e0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28f4011e50 con 0x7f29041085e0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290419c350 con 0x7f29041085e0 2024-03-28T18:28:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4019420 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290419f040 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f28f4019a80 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.962+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f28f4024730 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.963+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2904069250 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:03.965+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f28f4003a50 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.031+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f290419f480 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.032+0000 7f28faffd700 1 -- 172.21.15.145:0/702027234 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f28f4032030 con 0x7f29041085e0 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 msgr2=0x7f290419bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 --2- 172.21.15.145:0/702027234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fb0 tx=0x7f28f4011bf0 comp rx=0 tx=0).stop 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 shutdown_connections 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 --2- 172.21.15.145:0/702027234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29041085e0 0x7f290419bb80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 >> 172.21.15.145:0/702027234 conn(0x7f2904101f90 msgr2=0x7f2904102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 shutdown_connections 2024-03-28T18:28:04.979 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:04.033+0000 7f290ba68700 1 -- 172.21.15.145:0/702027234 wait complete. 2024-03-28T18:28:04.980 INFO:teuthology.orchestra.run.smithi145.stdout:Restarting the monitor... 2024-03-28T18:28:05.240 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:04 smithi145 systemd[1]: Stopping Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:28:05.505 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a[31773]: 2024-03-28T18:28:05.239+0000 7f86ceac9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:28:05.505 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a[31773]: 2024-03-28T18:28:05.239+0000 7f86ceac9700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T18:28:05.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:05 smithi145 podman[32023]: 2024-03-28 18:28:05.505788843 +0000 UTC m=+0.366788690 container died 0cf2f92da4d621cc085bae236a009dbfe282fa7662b835328a5cc9b9042813ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:28:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:05 smithi145 podman[32023]: 2024-03-28 18:28:05.856141327 +0000 UTC m=+0.717141180 container cleanup 0cf2f92da4d621cc085bae236a009dbfe282fa7662b835328a5cc9b9042813ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:28:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:05 smithi145 bash[32023]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a 2024-03-28T18:28:06.484 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 podman[32037]: 2024-03-28 18:28:06.150142503 +0000 UTC m=+0.645003486 container remove 0cf2f92da4d621cc085bae236a009dbfe282fa7662b835328a5cc9b9042813ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T18:28:06.484 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T18:28:06.484 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 systemd[1]: Stopped Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:28:06.484 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 systemd[1]: Starting Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:28:06.744 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 podman[32133]: 2024-03-28 18:28:06.387518802 +0000 UTC m=+0.018026988 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:28:06.744 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 podman[32133]: 2024-03-28 18:28:06.501565967 +0000 UTC m=+0.132074144 container create 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T18:28:06.745 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 podman[32133]: 2024-03-28 18:28:06.719047628 +0000 UTC m=+0.349555805 container init 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:28:06.745 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 podman[32133]: 2024-03-28 18:28:06.721862482 +0000 UTC m=+0.352370651 container start 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:28:06.805 INFO:teuthology.orchestra.run.smithi145.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: pidfile_write: ignore empty --pid-file 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: load: jerasure load: lrc 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: RocksDB version: 7.9.2 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Git sha 0 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T18:28:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: DB SUMMARY 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: DB Session ID: NJGU050WRC11TER9777J 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: CURRENT file: CURRENT 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100218 ; 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.error_if_exists: 0 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.create_if_missing: 0 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.paranoid_checks: 1 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.env: 0x55dd00940b20 2024-03-28T18:28:07.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.info_log: 0x55dd014bd920 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.statistics: (nil) 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.use_fsync: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_log_file_size: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_fallocate: 1 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T18:28:07.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.use_direct_reads: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.db_log_dir: 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.wal_dir: 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.write_buffer_manager: 0x55dd014901e0 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T18:28:07.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.unordered_write: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.row_cache: None 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.wal_filter: None 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.two_write_queues: 0 2024-03-28T18:28:07.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.wal_compression: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.atomic_flush: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.log_readahead_size: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T18:28:07.032 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T18:28:07.033 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_background_jobs: 2 2024-03-28T18:28:07.033 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_background_compactions: -1 2024-03-28T18:28:07.033 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_subcompactions: 1 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_open_files: -1 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_background_flushes: -1 2024-03-28T18:28:07.034 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Compression algorithms supported: 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kZSTD supported: 0 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kXpressCompression supported: 0 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kZlibCompression supported: 1 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kSnappyCompression supported: 1 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kLZ4Compression supported: 1 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: kBZip2Compression supported: 0 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.merge_operator: 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_filter: None 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T18:28:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dd014bda20) 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_type: 0 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_index_type: 0 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_shortening: 1 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: checksum: 4 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: no_block_cache: 0 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache: 0x55dd014c3090 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-03-28T18:28:07.036 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_options: 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: capacity : 536870912 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: num_shard_bits : 4 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: strict_capacity_limit : 0 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_cache_compressed: (nil) 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: persistent_cache: (nil) 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size: 4096 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_size_deviation: 10 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_restart_interval: 16 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: index_block_restart_interval: 1 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: metadata_block_size: 4096 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: partition_filters: 0 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: use_delta_encoding: 1 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: filter_policy: bloomfilter 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: whole_key_filtering: 1 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: verify_compression: 0 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-03-28T18:28:07.037 INFO:journalctl@ceph.mon.a.smithi145.stdout: format_version: 5 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_index_compression: 1 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: block_align: 0 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: max_auto_readahead_size: 262144 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: prepopulate_block_cache: 0 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: initial_auto_readahead_size: 8192 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression: NoCompression 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.num_levels: 7 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T18:28:07.038 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T18:28:07.039 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T18:28:07.040 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T18:28:07.041 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.table_properties_collectors: 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.inplace_update_support: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.bloom_locality: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.max_successive_merges: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.ttl: 2592000 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T18:28:07.042 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enable_blob_files: false 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.min_blob_size: 0 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 39fa40dd-dd8f-4f54-b304-7181f8b462df 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650486751052, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T18:28:07.043 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650486752044, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95419, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 93405, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13801, "raw_average_key_size": 50, "raw_value_size": 85878, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711650486, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "39fa40dd-dd8f-4f54-b304-7181f8b462df", "db_session_id": "NJGU050WRC11TER9777J", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650486752125, "job": 1, "event": "recovery_finished"} 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dd015b6000 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: DB pointer 0x55dd015a0000 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** DB Stats ** 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.044 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: L0 2/0 95.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Sum 2/0 95.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** Compaction Stats [default] ** 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T18:28:07.045 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: Cumulative compaction: 0.00 GB write, 30.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: Interval compaction: 0.00 GB write, 30.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: Block cache BinnedLRUCache@0x55dd014c3090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: starting mon.a rank 0 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???) e1 preinit fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).mds e1 new map 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).mds e1 print_map 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: e1 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:28:07.046 INFO:journalctl@ceph.mon.a.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout: legacy client fscid: -1 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout: 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout: No filesystems configured 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 bash[32133]: 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 systemd[1]: Started Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: monmap e1: 1 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: fsmap 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T18:28:07.047 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:06 smithi145 ceph-mon[32151]: mgrmap e1: no daemons active 2024-03-28T18:28:07.873 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.228+0000 7ff61ba95700 1 Processor -- start 2024-03-28T18:28:07.873 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.228+0000 7ff61ba95700 1 -- start start 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.228+0000 7ff61ba95700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff61ba95700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff614107370 con 0x7ff614106980 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff619831700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff619831700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39364/0 (socket says 172.21.15.145:39364) 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff619831700 1 -- 172.21.15.145:0/1672925229 learned_addr learned my addr 172.21.15.145:0/1672925229 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff619831700 1 -- 172.21.15.145:0/1672925229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614107b80 con 0x7ff614106980 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff619831700 1 --2- 172.21.15.145:0/1672925229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff61000bb40 tx=0x7ff61000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89716ee2c6c7d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.229+0000 7ff61882f700 1 -- 172.21.15.145:0/1672925229 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff61000dbe0 con 0x7ff614106980 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61882f700 1 -- 172.21.15.145:0/1672925229 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6100044f0 con 0x7ff614106980 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61882f700 1 -- 172.21.15.145:0/1672925229 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff610010470 con 0x7ff614106980 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 -- 172.21.15.145:0/1672925229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 msgr2=0x7ff614106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:07.874 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 --2- 172.21.15.145:0/1672925229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff61000bb40 tx=0x7ff61000be50 comp rx=0 tx=0).stop 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 -- 172.21.15.145:0/1672925229 shutdown_connections 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 --2- 172.21.15.145:0/1672925229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614106da0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 -- 172.21.15.145:0/1672925229 >> 172.21.15.145:0/1672925229 conn(0x7ff614101f60 msgr2=0x7ff614104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 -- 172.21.15.145:0/1672925229 shutdown_connections 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.230+0000 7ff61ba95700 1 -- 172.21.15.145:0/1672925229 wait complete. 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff61ba95700 1 Processor -- start 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff61ba95700 1 -- start start 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff61ba95700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff61ba95700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61407aff0 con 0x7ff614106980 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff619831700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff619831700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39368/0 (socket says 172.21.15.145:39368) 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff619831700 1 -- 172.21.15.145:0/2861876115 learned_addr learned my addr 172.21.15.145:0/2861876115 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.231+0000 7ff619831700 1 -- 172.21.15.145:0/2861876115 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614199dc0 con 0x7ff614106980 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff619831700 1 --2- 172.21.15.145:0/2861876115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b340 tx=0x7ff6100040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:07.875 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff610027070 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61419a050 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff610010be0 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff610022c40 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61419cd40 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff61002a070 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.232+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff61001f070 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.233+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6140637c0 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.235+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff610031030 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.301+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ff614192de0 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.304+0000 7ff60affd700 1 -- 172.21.15.145:0/2861876115 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff610015070 con 0x7ff614106980 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 msgr2=0x7ff614199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 --2- 172.21.15.145:0/2861876115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b340 tx=0x7ff6100040b0 comp rx=0 tx=0).stop 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 shutdown_connections 2024-03-28T18:28:07.876 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 --2- 172.21.15.145:0/2861876115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff614106980 0x7ff614199880 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:07.877 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 >> 172.21.15.145:0/2861876115 conn(0x7ff614101f60 msgr2=0x7ff61418fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:07.877 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 shutdown_connections 2024-03-28T18:28:07.877 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:07.305+0000 7ff61ba95700 1 -- 172.21.15.145:0/2861876115 wait complete. 2024-03-28T18:28:07.877 INFO:teuthology.orchestra.run.smithi145.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T18:28:07.891 INFO:teuthology.orchestra.run.smithi145.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:28:07.891 INFO:teuthology.orchestra.run.smithi145.stdout:Creating mgr... 2024-03-28T18:28:07.891 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T18:28:07.892 INFO:teuthology.orchestra.run.smithi145.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T18:28:08.139 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x 2024-03-28T18:28:08.139 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T18:28:08.287 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715.target.wants/ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-b991dbde-ed30-11ee-95d0-87774f69a715@.service. 2024-03-28T18:28:08.518 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:08 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2861876115' entity='client.admin' 2024-03-28T18:28:08.518 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 systemd[1]: Starting Ceph mgr.x for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:28:08.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 podman[32366]: 2024-03-28 18:28:08.518816239 +0000 UTC m=+0.107318128 container create f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T18:28:08.777 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 podman[32366]: 2024-03-28 18:28:08.429208281 +0000 UTC m=+0.017710176 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:28:08.777 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 podman[32366]: 2024-03-28 18:28:08.712696009 +0000 UTC m=+0.301197900 container init f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:28:08.777 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 podman[32366]: 2024-03-28 18:28:08.715279602 +0000 UTC m=+0.303781490 container start f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:28:08.826 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:28:08.826 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout disabled 2024-03-28T18:28:08.833 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:28:08.833 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:28:08.833 INFO:teuthology.orchestra.run.smithi145.stdout:firewalld.service is not enabled 2024-03-28T18:28:08.833 INFO:teuthology.orchestra.run.smithi145.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T18:28:08.841 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:28:08.841 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout disabled 2024-03-28T18:28:08.848 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:28:08.848 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:28:08.849 INFO:teuthology.orchestra.run.smithi145.stdout:firewalld.service is not enabled 2024-03-28T18:28:08.849 INFO:teuthology.orchestra.run.smithi145.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T18:28:08.849 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mgr to start... 2024-03-28T18:28:08.849 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mgr... 2024-03-28T18:28:09.063 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 bash[32366]: f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 2024-03-28T18:28:09.063 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:08 smithi145 systemd[1]: Started Ceph mgr.x for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:28:09.359 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:09.262+0000 7f8819050200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:28:09.359 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:09.359+0000 7f8819050200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:28:09.668 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1529404859' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:28:09.941 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:09.942 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:28:09.943 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:28:09.943 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:28:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:09.951 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:28:01.180614+0000", 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.291+0000 7f4b8a44f700 1 Processor -- start 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.291+0000 7f4b8a44f700 1 -- start start 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.291+0000 7f4b8a44f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.291+0000 7f4b8a44f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84106b00 con 0x7f4b84106110 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b83fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b83fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39388/0 (socket says 172.21.15.145:39388) 2024-03-28T18:28:09.952 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b83fff700 1 -- 172.21.15.145:0/2211373426 learned_addr learned my addr 172.21.15.145:0/2211373426 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b83fff700 1 -- 172.21.15.145:0/2211373426 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b84107310 con 0x7f4b84106110 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b83fff700 1 --2- 172.21.15.145:0/2211373426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800bb40 tx=0x7f4b7800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e3edfdc4ae880f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b82ffd700 1 -- 172.21.15.145:0/2211373426 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b7800dad0 con 0x7f4b84106110 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b82ffd700 1 -- 172.21.15.145:0/2211373426 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b780044f0 con 0x7f4b84106110 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.292+0000 7f4b82ffd700 1 -- 172.21.15.145:0/2211373426 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78017410 con 0x7f4b84106110 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 -- 172.21.15.145:0/2211373426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 msgr2=0x7f4b84106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 --2- 172.21.15.145:0/2211373426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800bb40 tx=0x7f4b7800be50 comp rx=0 tx=0).stop 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 -- 172.21.15.145:0/2211373426 shutdown_connections 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 --2- 172.21.15.145:0/2211373426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b84106530 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 -- 172.21.15.145:0/2211373426 >> 172.21.15.145:0/2211373426 conn(0x7f4b841016f0 msgr2=0x7f4b84103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 -- 172.21.15.145:0/2211373426 shutdown_connections 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.293+0000 7f4b8a44f700 1 -- 172.21.15.145:0/2211373426 wait complete. 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b8a44f700 1 Processor -- start 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b8a44f700 1 -- start start 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b8a44f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b8a44f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84107a90 con 0x7f4b84106110 2024-03-28T18:28:09.953 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b83fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b83fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39404/0 (socket says 172.21.15.145:39404) 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b83fff700 1 -- 172.21.15.145:0/1529404859 learned_addr learned my addr 172.21.15.145:0/1529404859 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b83fff700 1 -- 172.21.15.145:0/1529404859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b84199e30 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.294+0000 7f4b83fff700 1 --2- 172.21.15.145:0/1529404859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800bb10 tx=0x7f4b78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78004090 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b78004230 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b78021b90 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8419a0c0 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8419cdb0 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4b7802a400 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.295+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b7801f7a0 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.296+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b84069250 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.298+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4b78012070 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.371+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4b8402dca0 con 0x7f4b84106110 2024-03-28T18:28:09.954 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.372+0000 7f4b817fa700 1 -- 172.21.15.145:0/1529404859 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4b78021400 con 0x7f4b84106110 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 msgr2=0x7f4b841998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 --2- 172.21.15.145:0/1529404859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800bb10 tx=0x7f4b78004750 comp rx=0 tx=0).stop 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 shutdown_connections 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 --2- 172.21.15.145:0/1529404859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b84106110 0x7f4b841998f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 >> 172.21.15.145:0/1529404859 conn(0x7f4b841016f0 msgr2=0x7f4b84190560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 shutdown_connections 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:09.373+0000 7f4b8a44f700 1 -- 172.21.15.145:0/1529404859 wait complete. 2024-03-28T18:28:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:mgr not available, waiting (1/15)... 2024-03-28T18:28:10.249 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.010+0000 7f8819050200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:28:10.249 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.103+0000 7f8819050200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:28:10.249 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.249+0000 7f8819050200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:28:10.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.343+0000 7f8819050200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:28:10.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.423+0000 7f8819050200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:28:10.967 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.682+0000 7f8819050200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:28:10.967 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.789+0000 7f8819050200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:10.967 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.874+0000 7f8819050200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:28:10.967 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:10.967+0000 7f8819050200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:28:11.276 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:11.149+0000 7f8819050200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:28:12.347 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:12.112+0000 7f8819050200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:28:12.347 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:12.245+0000 7f8819050200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:28:12.609 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3615928063' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:28:12.889 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:12.609+0000 7f8819050200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:28:12.889 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:12.697+0000 7f8819050200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:28:12.890 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:12.889+0000 7f8819050200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:13.004 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:28:13.005 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:28:13.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:28:13.014 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.015 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:28:01.180614+0000", 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f204a7ef700 1 Processor -- start 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f204a7ef700 1 -- start start 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f204a7ef700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f204a7ef700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204407c910 con 0x7f204407dee0 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f2043fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f2043fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39420/0 (socket says 172.21.15.145:39420) 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.362+0000 7f2043fff700 1 -- 172.21.15.145:0/3875935044 learned_addr learned my addr 172.21.15.145:0/3875935044 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.363+0000 7f2043fff700 1 -- 172.21.15.145:0/3875935044 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204407ca90 con 0x7f204407dee0 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.363+0000 7f2043fff700 1 --2- 172.21.15.145:0/3875935044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20380098a0 tx=0x7f2038009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd30afa09a2cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:13.016 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.363+0000 7f2042ffd700 1 -- 172.21.15.145:0/3875935044 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2038004160 con 0x7f204407dee0 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.363+0000 7f2042ffd700 1 -- 172.21.15.145:0/3875935044 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2038004300 con 0x7f204407dee0 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.363+0000 7f2042ffd700 1 -- 172.21.15.145:0/3875935044 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2038011b50 con 0x7f204407dee0 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- 172.21.15.145:0/3875935044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 msgr2=0x7f204407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 --2- 172.21.15.145:0/3875935044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20380098a0 tx=0x7f2038009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- 172.21.15.145:0/3875935044 shutdown_connections 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 --2- 172.21.15.145:0/3875935044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204407c340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- 172.21.15.145:0/3875935044 >> 172.21.15.145:0/3875935044 conn(0x7f2044103070 msgr2=0x7f2044105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- 172.21.15.145:0/3875935044 shutdown_connections 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- 172.21.15.145:0/3875935044 wait complete. 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 Processor -- start 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.364+0000 7f204a7ef700 1 -- start start 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f204a7ef700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f204a7ef700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204410e080 con 0x7f204407dee0 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f2043fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f2043fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39434/0 (socket says 172.21.15.145:39434) 2024-03-28T18:28:13.017 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f2043fff700 1 -- 172.21.15.145:0/3615928063 learned_addr learned my addr 172.21.15.145:0/3615928063 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f2043fff700 1 -- 172.21.15.145:0/3615928063 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204419e1f0 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f2043fff700 1 --2- 172.21.15.145:0/3615928063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2038011850 tx=0x7f2038011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.365+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2038003af0 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204419e480 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2038020070 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20380195a0 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20441a1170 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2038019c30 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2038011e00 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.366+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2044069250 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.369+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2038003c90 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.441+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f20441a1340 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.442+0000 7f20417fa700 1 -- 172.21.15.145:0/3615928063 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f203801a430 con 0x7f204407dee0 2024-03-28T18:28:13.018 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 msgr2=0x7f204419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 --2- 172.21.15.145:0/3615928063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2038011850 tx=0x7f2038011c10 comp rx=0 tx=0).stop 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 shutdown_connections 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 --2- 172.21.15.145:0/3615928063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f204407dee0 0x7f204419dcb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 >> 172.21.15.145:0/3615928063 conn(0x7f2044103070 msgr2=0x7f2044103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 shutdown_connections 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:12.443+0000 7f204a7ef700 1 -- 172.21.15.145:0/3615928063 wait complete. 2024-03-28T18:28:13.019 INFO:teuthology.orchestra.run.smithi145.stdout:mgr not available, waiting (2/15)... 2024-03-28T18:28:13.276 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:13 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:13.072+0000 7f8819050200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:28:13.604 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:13 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:13.347+0000 7f8819050200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:13.604 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:13 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:13.603+0000 7f8819050200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:14.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:13 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:13.842+0000 7f8819050200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:28:15.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:14 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:14.619+0000 7f8819050200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:28:15.403 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:15.193+0000 7f8819050200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:28:15.403 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:15.293+0000 7f8819050200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:28:15.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:15.520+0000 7f8819050200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:28:15.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4232288473' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:28:15.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: Activating manager daemon x 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: mgrmap e2: x(active, starting, since 0.00295748s) 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:28:15.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:15 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:28:16.111 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:16.112 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.113 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.114 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:28:01.180614+0000", 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:28:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.409+0000 7fd09ae26700 1 Processor -- start 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.409+0000 7fd09ae26700 1 -- start start 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.409+0000 7fd09ae26700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.409+0000 7fd09ae26700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09407bed0 con 0x7fd0941084d0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.409+0000 7fd098bc2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd098bc2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38412/0 (socket says 172.21.15.145:38412) 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd098bc2700 1 -- 172.21.15.145:0/1169385193 learned_addr learned my addr 172.21.15.145:0/1169385193 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd098bc2700 1 -- 172.21.15.145:0/1169385193 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09410ae00 con 0x7fd0941084d0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd098bc2700 1 --2- 172.21.15.145:0/1169385193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098a0 tx=0x7fd088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445e21686e4f2cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd0937fe700 1 -- 172.21.15.145:0/1169385193 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd088004160 con 0x7fd0941084d0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd0937fe700 1 -- 172.21.15.145:0/1169385193 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd088004300 con 0x7fd0941084d0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.410+0000 7fd0937fe700 1 -- 172.21.15.145:0/1169385193 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd088011b50 con 0x7fd0941084d0 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 -- 172.21.15.145:0/1169385193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 msgr2=0x7fd09410a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 --2- 172.21.15.145:0/1169385193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098a0 tx=0x7fd088009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:16.116 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 -- 172.21.15.145:0/1169385193 shutdown_connections 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 --2- 172.21.15.145:0/1169385193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09410a8c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 -- 172.21.15.145:0/1169385193 >> 172.21.15.145:0/1169385193 conn(0x7fd094101f20 msgr2=0x7fd094104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 -- 172.21.15.145:0/1169385193 shutdown_connections 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 -- 172.21.15.145:0/1169385193 wait complete. 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.411+0000 7fd09ae26700 1 Processor -- start 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd09ae26700 1 -- start start 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd09ae26700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd09ae26700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09410be30 con 0x7fd0941084d0 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd098bc2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd098bc2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38414/0 (socket says 172.21.15.145:38414) 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd098bc2700 1 -- 172.21.15.145:0/4232288473 learned_addr learned my addr 172.21.15.145:0/4232288473 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd098bc2700 1 -- 172.21.15.145:0/4232288473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09419bf70 con 0x7fd0941084d0 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd098bc2700 1 --2- 172.21.15.145:0/4232288473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd088009fb0 tx=0x7fd088011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0880038b0 con 0x7fd0941084d0 2024-03-28T18:28:16.117 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd088011e50 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.412+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09419c200 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.413+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd088019420 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.413+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09419eef0 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.413+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd088019a60 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.413+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd088024730 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.414+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd094069250 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.416+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd088043dd0 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.488+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd09419f490 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.488+0000 7fd091ffb700 1 -- 172.21.15.145:0/4232288473 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd088032030 con 0x7fd0941084d0 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 msgr2=0x7fd09419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 --2- 172.21.15.145:0/4232288473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd088009fb0 tx=0x7fd088011bf0 comp rx=0 tx=0).stop 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 shutdown_connections 2024-03-28T18:28:16.118 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 --2- 172.21.15.145:0/4232288473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0941084d0 0x7fd09419ba30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:16.119 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 >> 172.21.15.145:0/4232288473 conn(0x7fd094101f20 msgr2=0x7fd094102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:16.119 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 shutdown_connections 2024-03-28T18:28:16.119 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:15.490+0000 7fd09ae26700 1 -- 172.21.15.145:0/4232288473 wait complete. 2024-03-28T18:28:16.119 INFO:teuthology.orchestra.run.smithi145.stdout:mgr not available, waiting (3/15)... 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: Manager daemon x is now available 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' 2024-03-28T18:28:17.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:16 smithi145 ceph-mon[32151]: from='mgr.14100 172.21.15.145:0/2543811947' entity='mgr.x' 2024-03-28T18:28:18.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:17 smithi145 ceph-mon[32151]: mgrmap e3: x(active, since 1.01879s) 2024-03-28T18:28:18.837 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:18 smithi145 ceph-mon[32151]: mgrmap e4: x(active, since 2s) 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:28:19.351 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:28:19.352 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:28:19.353 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:28:19.354 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:28:01.180614+0000", 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:28:19.355 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.555+0000 7f0d2ceb7700 1 Processor -- start 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.555+0000 7f0d2ceb7700 1 -- start start 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2ceb7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2ceb7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2807bed0 con 0x7f0d281084d0 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38498/0 (socket says 172.21.15.145:38498) 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2659c700 1 -- 172.21.15.145:0/2881320970 learned_addr learned my addr 172.21.15.145:0/2881320970 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2659c700 1 -- 172.21.15.145:0/2881320970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2810ae00 con 0x7f0d281084d0 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2659c700 1 --2- 172.21.15.145:0/2881320970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0d180098a0 tx=0x7f0d18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66efd5ca7bbda7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:19.356 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2559a700 1 -- 172.21.15.145:0/2881320970 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d18004160 con 0x7f0d281084d0 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2559a700 1 -- 172.21.15.145:0/2881320970 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d18004300 con 0x7f0d281084d0 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.556+0000 7f0d2559a700 1 -- 172.21.15.145:0/2881320970 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d18011b50 con 0x7f0d281084d0 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/2881320970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 msgr2=0x7f0d2810a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/2881320970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0d180098a0 tx=0x7f0d18009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/2881320970 shutdown_connections 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/2881320970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d2810a8c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/2881320970 >> 172.21.15.145:0/2881320970 conn(0x7f0d28101f20 msgr2=0x7f0d28104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/2881320970 shutdown_connections 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.557+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/2881320970 wait complete. 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2ceb7700 1 Processor -- start 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2ceb7700 1 -- start start 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2ceb7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2ceb7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2810be30 con 0x7f0d281084d0 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38500/0 (socket says 172.21.15.145:38500) 2024-03-28T18:28:19.357 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2659c700 1 -- 172.21.15.145:0/1473925399 learned_addr learned my addr 172.21.15.145:0/1473925399 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.558+0000 7f0d2659c700 1 -- 172.21.15.145:0/1473925399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d28108290 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d2659c700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d18011b60 tx=0x7f0d18011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d18003970 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d18003b10 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d28106480 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d18024610 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d281069c0 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.559+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f0d180248c0 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.560+0000 7f0d177fe700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 0x7f0d1003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.560+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d28069250 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.560+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d18011e00 con 0x7f0d281084d0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.560+0000 7f0d25d9b700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 0x7f0d1003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:19.358 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.560+0000 7f0d25d9b700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 0x7f0d1003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c006fd0 tx=0x7f0d1c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.565+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0d1800ba30 con 0x7f0d281084d0 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.836+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0d281072f0 con 0x7f0d281084d0 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.837+0000 7f0d177fe700 1 -- 172.21.15.145:0/1473925399 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f0d1804c9c0 con 0x7f0d281084d0 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.839+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 msgr2=0x7f0d1003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 0x7f0d1003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c006fd0 tx=0x7f0d1c006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 msgr2=0x7f0d28107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d18011b60 tx=0x7f0d18011c40 comp rx=0 tx=0).stop 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 shutdown_connections 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f0d10038f20 0x7f0d1003b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 --2- 172.21.15.145:0/1473925399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d281084d0 0x7f0d28107d50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 >> 172.21.15.145:0/1473925399 conn(0x7f0d28101f20 msgr2=0x7f0d28102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 shutdown_connections 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:18.840+0000 7f0d2ceb7700 1 -- 172.21.15.145:0/1473925399 wait complete. 2024-03-28T18:28:19.359 INFO:teuthology.orchestra.run.smithi145.stdout:mgr is available 2024-03-28T18:28:19.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:19 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1473925399' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout fsid = b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:20.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5038221700 1 Processor -- start 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5038221700 1 -- start start 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5038221700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5038221700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503007c910 con 0x7f503007de90 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5035fbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:20.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5035fbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38508/0 (socket says 172.21.15.145:38508) 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.738+0000 7f5035fbd700 1 -- 172.21.15.145:0/979573735 learned_addr learned my addr 172.21.15.145:0/979573735 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5035fbd700 1 -- 172.21.15.145:0/979573735 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503007ca90 con 0x7f503007de90 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5035fbd700 1 --2- 172.21.15.145:0/979573735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f502c0098a0 tx=0x7f502c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e2ac921bd5f7ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5034fbb700 1 -- 172.21.15.145:0/979573735 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f502c004160 con 0x7f503007de90 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5034fbb700 1 -- 172.21.15.145:0/979573735 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f502c004300 con 0x7f503007de90 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5034fbb700 1 -- 172.21.15.145:0/979573735 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f502c011b50 con 0x7f503007de90 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5038221700 1 -- 172.21.15.145:0/979573735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 msgr2=0x7f503007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.739+0000 7f5038221700 1 --2- 172.21.15.145:0/979573735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f502c0098a0 tx=0x7f502c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- 172.21.15.145:0/979573735 shutdown_connections 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 --2- 172.21.15.145:0/979573735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503007c340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- 172.21.15.145:0/979573735 >> 172.21.15.145:0/979573735 conn(0x7f5030103130 msgr2=0x7f5030105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- 172.21.15.145:0/979573735 shutdown_connections 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- 172.21.15.145:0/979573735 wait complete. 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 Processor -- start 2024-03-28T18:28:20.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- start start 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.740+0000 7f5038221700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503010e0e0 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5035fbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5035fbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38520/0 (socket says 172.21.15.145:38520) 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5035fbd700 1 -- 172.21.15.145:0/1228222923 learned_addr learned my addr 172.21.15.145:0/1228222923 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5035fbd700 1 -- 172.21.15.145:0/1228222923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503019e200 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5035fbd700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f502c009fb0 tx=0x7f502c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f502c0038b0 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f502c011e50 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503019e490 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f502c019420 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.741+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50301a1180 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.742+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f502c0196d0 con 0x7f503007de90 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.742+0000 7f5026ffd700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 0x7f501c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:20.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.742+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f502c04cfe0 con 0x7f503007de90 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.742+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5030069250 con 0x7f503007de90 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.742+0000 7f50357bc700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 0x7f501c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.743+0000 7f50357bc700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 0x7f501c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5020006fd0 tx=0x7f5020006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.747+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f502c00bc70 con 0x7f503007de90 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.939+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f50301a1350 con 0x7f503007de90 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.940+0000 7f5026ffd700 1 -- 172.21.15.145:0/1228222923 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f502c003a50 con 0x7f503007de90 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.942+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 msgr2=0x7f501c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.942+0000 7f5038221700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 0x7f501c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5020006fd0 tx=0x7f5020006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.942+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 msgr2=0x7f503019dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.942+0000 7f5038221700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f502c009fb0 tx=0x7f502c011bf0 comp rx=0 tx=0).stop 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 shutdown_connections 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f501c038ed0 0x7f501c03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 --2- 172.21.15.145:0/1228222923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503007de90 0x7f503019dcc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 >> 172.21.15.145:0/1228222923 conn(0x7f5030103130 msgr2=0x7f5030103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:20.482 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 shutdown_connections 2024-03-28T18:28:20.483 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:19.943+0000 7f5038221700 1 -- 172.21.15.145:0/1228222923 wait complete. 2024-03-28T18:28:20.483 INFO:teuthology.orchestra.run.smithi145.stdout:Enabling cephadm module... 2024-03-28T18:28:20.813 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:20 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1228222923' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T18:28:21.884 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:21 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1064558245' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T18:28:21.884 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:21 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: ignoring --setuser ceph since I am not root 2024-03-28T18:28:21.884 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:21 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: ignoring --setgroup ceph since I am not root 2024-03-28T18:28:22.142 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:22.130+0000 7f2916b2c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:28:22.149 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.829+0000 7f89afb31700 1 Processor -- start 2024-03-28T18:28:22.149 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89afb31700 1 -- start start 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89afb31700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89afb31700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a8108470 con 0x7f89a8107a80 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89ad8cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89ad8cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38536/0 (socket says 172.21.15.145:38536) 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89ad8cd700 1 -- 172.21.15.145:0/84137728 learned_addr learned my addr 172.21.15.145:0/84137728 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89ad8cd700 1 -- 172.21.15.145:0/84137728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a8108c80 con 0x7f89a8107a80 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.830+0000 7f89ad8cd700 1 --2- 172.21.15.145:0/84137728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89980098a0 tx=0x7f8998009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b111c00044648477 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89ac8cb700 1 -- 172.21.15.145:0/84137728 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8998004160 con 0x7f89a8107a80 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89ac8cb700 1 -- 172.21.15.145:0/84137728 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8998004300 con 0x7f89a8107a80 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89ac8cb700 1 -- 172.21.15.145:0/84137728 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8998011b50 con 0x7f89a8107a80 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 -- 172.21.15.145:0/84137728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 msgr2=0x7f89a8107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 --2- 172.21.15.145:0/84137728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89980098a0 tx=0x7f8998009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 -- 172.21.15.145:0/84137728 shutdown_connections 2024-03-28T18:28:22.150 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 --2- 172.21.15.145:0/84137728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a8107ea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 -- 172.21.15.145:0/84137728 >> 172.21.15.145:0/84137728 conn(0x7f89a81030c0 msgr2=0x7f89a81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 -- 172.21.15.145:0/84137728 shutdown_connections 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.831+0000 7f89afb31700 1 -- 172.21.15.145:0/84137728 wait complete. 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89afb31700 1 Processor -- start 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89afb31700 1 -- start start 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89afb31700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89afb31700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a810c030 con 0x7f89a8107a80 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89ad8cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89ad8cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38546/0 (socket says 172.21.15.145:38546) 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89ad8cd700 1 -- 172.21.15.145:0/1064558245 learned_addr learned my addr 172.21.15.145:0/1064558245 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89ad8cd700 1 -- 172.21.15.145:0/1064558245 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a819bf90 con 0x7f89a8107a80 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.832+0000 7f89ad8cd700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8998011670 tx=0x7f8998011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89980039c0 con 0x7f89a8107a80 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8998011e50 con 0x7f89a8107a80 2024-03-28T18:28:22.151 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f899801a5c0 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a819c220 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a819ef10 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.833+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f8998024430 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.834+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a80637c0 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.834+0000 7f899effd700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 0x7f899403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.834+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f899804d160 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.834+0000 7f89ad0cc700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 0x7f899403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.834+0000 7f89ad0cc700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 0x7f899403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89a4006fd0 tx=0x7f89a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:20.839+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f899802d910 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.087+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f89a819f0e0 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.560+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f89a819f0e0 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.561+0000 7f899effd700 1 -- 172.21.15.145:0/1064558245 <== mon.0 v2:172.21.15.145:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f899802d3f0 con 0x7f89a8107a80 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 msgr2=0x7f899403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 0x7f899403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89a4006fd0 tx=0x7f89a4006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 msgr2=0x7f89a819ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:22.152 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8998011670 tx=0x7f8998011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 shutdown_connections 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f8994038ed0 0x7f899403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 --2- 172.21.15.145:0/1064558245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89a8107a80 0x7f89a819ba50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 >> 172.21.15.145:0/1064558245 conn(0x7f89a81030c0 msgr2=0x7f89a8103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.563+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 shutdown_connections 2024-03-28T18:28:22.153 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:21.564+0000 7f89afb31700 1 -- 172.21.15.145:0/1064558245 wait complete. 2024-03-28T18:28:22.394 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:22.234+0000 7f2916b2c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:28:22.774 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1064558245' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T18:28:22.775 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-mon[32151]: mgrmap e5: x(active, since 6s) 2024-03-28T18:28:23.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:22.899+0000 7f2916b2c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:28:23.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:22 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:22.991+0000 7f2916b2c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:28:23.424 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.135+0000 7f2916b2c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:28:23.424 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.231+0000 7f2916b2c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:28:23.424 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.315+0000 7f2916b2c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb7084700 1 Processor -- start 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb7084700 1 -- start start 2024-03-28T18:28:23.433 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb7084700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb7084700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0107360 con 0x7fbdb0106970 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb4e20700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb4e20700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58064/0 (socket says 172.21.15.145:58064) 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb4e20700 1 -- 172.21.15.145:0/3879823568 learned_addr learned my addr 172.21.15.145:0/3879823568 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.569+0000 7fbdb4e20700 1 -- 172.21.15.145:0/3879823568 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0107b70 con 0x7fbdb0106970 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdb4e20700 1 --2- 172.21.15.145:0/3879823568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbda400bb40 tx=0x7fbda400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b9eb91e34b390b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdaf7fe700 1 -- 172.21.15.145:0/3879823568 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda400dbe0 con 0x7fbdb0106970 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdaf7fe700 1 -- 172.21.15.145:0/3879823568 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbda40044f0 con 0x7fbdb0106970 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdaf7fe700 1 -- 172.21.15.145:0/3879823568 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda4010470 con 0x7fbdb0106970 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdb7084700 1 -- 172.21.15.145:0/3879823568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 msgr2=0x7fbdb0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdb7084700 1 --2- 172.21.15.145:0/3879823568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbda400bb40 tx=0x7fbda400be50 comp rx=0 tx=0).stop 2024-03-28T18:28:23.434 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.570+0000 7fbdb7084700 1 -- 172.21.15.145:0/3879823568 shutdown_connections 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 --2- 172.21.15.145:0/3879823568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb0106d90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 -- 172.21.15.145:0/3879823568 >> 172.21.15.145:0/3879823568 conn(0x7fbdb0101f10 msgr2=0x7fbdb0104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 -- 172.21.15.145:0/3879823568 shutdown_connections 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 -- 172.21.15.145:0/3879823568 wait complete. 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 Processor -- start 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 -- start start 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb7084700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb007aff0 con 0x7fbdb0106970 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb4e20700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb4e20700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58066/0 (socket says 172.21.15.145:58066) 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.571+0000 7fbdb4e20700 1 -- 172.21.15.145:0/195978373 learned_addr learned my addr 172.21.15.145:0/195978373 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdb4e20700 1 -- 172.21.15.145:0/195978373 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb019e1d0 con 0x7fbdb0106970 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdb4e20700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbda400b340 tx=0x7fbda400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda4010680 con 0x7fbdb0106970 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb019e460 con 0x7fbdb0106970 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbda401d070 con 0x7fbdb0106970 2024-03-28T18:28:23.435 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbda4021af0 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.572+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb01a1150 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fbda401f030 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb0069250 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdadffb700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 0x7fbda003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdaffff700 1 -- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 msgr2=0x7fbda003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbda404dcb0 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.573+0000 7fbdaffff700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 0x7fbda003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.578+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbda4015070 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.774+0000 7fbdaffff700 1 -- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 msgr2=0x7fbda003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.774+0000 7fbdaffff700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 0x7fbda003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.839+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbdb01a1370 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.840+0000 7fbdadffb700 1 -- 172.21.15.145:0/195978373 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fbdb01a1370 con 0x7fbdb0106970 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 msgr2=0x7fbda003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 0x7fbda003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 msgr2=0x7fbdb019dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:23.436 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbda400b340 tx=0x7fbda400dd90 comp rx=0 tx=0).stop 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 shutdown_connections 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7fbda0038f20 0x7fbda003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 --2- 172.21.15.145:0/195978373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbdb0106970 0x7fbdb019dc90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 >> 172.21.15.145:0/195978373 conn(0x7fbdb0101f10 msgr2=0x7fbdb018fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 shutdown_connections 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:22.842+0000 7fbdb7084700 1 -- 172.21.15.145:0/195978373 wait complete. 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for the mgr to restart... 2024-03-28T18:28:23.437 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mgr epoch 5... 2024-03-28T18:28:23.695 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/195978373' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T18:28:23.695 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.583+0000 7f2916b2c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:28:23.695 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.694+0000 7f2916b2c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:24.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.781+0000 7f2916b2c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:28:24.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:23.881+0000 7f2916b2c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:28:24.458 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:24 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:24.055+0000 7f2916b2c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:28:25.259 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:24 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:24.997+0000 7f2916b2c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:28:25.259 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:25.123+0000 7f2916b2c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:28:25.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:25.452+0000 7f2916b2c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:28:25.912 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:25.539+0000 7f2916b2c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:28:25.912 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:25.730+0000 7f2916b2c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:25.912 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:25.911+0000 7f2916b2c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:28:26.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:26 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:26.190+0000 7f2916b2c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:26.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:26 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:26.447+0000 7f2916b2c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:26.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:26 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:26.679+0000 7f2916b2c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:28:27.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:27 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:27.463+0000 7f2916b2c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:28:28.333 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:28.027+0000 7f2916b2c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:28:28.333 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:28.122+0000 7f2916b2c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:28:28.333 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:28.332+0000 7f2916b2c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:28:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: Active manager daemon x restarted 2024-03-28T18:28:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: Activating manager daemon x 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: mgrmap e6: x(active, starting, since 0.00331885s) 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: Manager daemon x is now available 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:28 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:29.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: Found migration_current of "None". Setting to last migration. 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:29.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:29 smithi145 ceph-mon[32151]: mgrmap e7: x(active, since 1.00544s) 2024-03-28T18:28:29.914 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:29.914 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T18:28:29.914 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a2e529700 1 Processor -- start 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a2e529700 1 -- start start 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a2e529700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a2e529700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2807bed0 con 0x7f2a281084d0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a27fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a27fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58068/0 (socket says 172.21.15.145:58068) 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a27fff700 1 -- 172.21.15.145:0/3697136500 learned_addr learned my addr 172.21.15.145:0/3697136500 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a27fff700 1 -- 172.21.15.145:0/3697136500 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2810ae00 con 0x7f2a281084d0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.851+0000 7f2a27fff700 1 --2- 172.21.15.145:0/3697136500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a180098a0 tx=0x7f2a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b3187c094ecf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a26ffd700 1 -- 172.21.15.145:0/3697136500 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18004160 con 0x7f2a281084d0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a26ffd700 1 -- 172.21.15.145:0/3697136500 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a18004300 con 0x7f2a281084d0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a26ffd700 1 -- 172.21.15.145:0/3697136500 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18011b50 con 0x7f2a281084d0 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 -- 172.21.15.145:0/3697136500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 msgr2=0x7f2a2810a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:29.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 --2- 172.21.15.145:0/3697136500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a180098a0 tx=0x7f2a18009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 -- 172.21.15.145:0/3697136500 shutdown_connections 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 --2- 172.21.15.145:0/3697136500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a2810a8c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 -- 172.21.15.145:0/3697136500 >> 172.21.15.145:0/3697136500 conn(0x7f2a28101f20 msgr2=0x7f2a28104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 -- 172.21.15.145:0/3697136500 shutdown_connections 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.852+0000 7f2a2e529700 1 -- 172.21.15.145:0/3697136500 wait complete. 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a2e529700 1 Processor -- start 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a2e529700 1 -- start start 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a2e529700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a2e529700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2810be30 con 0x7f2a281084d0 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a27fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a27fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58080/0 (socket says 172.21.15.145:58080) 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.853+0000 7f2a27fff700 1 -- 172.21.15.145:0/556404804 learned_addr learned my addr 172.21.15.145:0/556404804 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a27fff700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28108290 con 0x7f2a281084d0 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a27fff700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009fb0 tx=0x7f2a18011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a180038b0 con 0x7f2a281084d0 2024-03-28T18:28:29.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a28106480 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a18011e50 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18019420 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.854+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a281069c0 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f2a180196f0 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a257fa700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a1003baf0 con 0x7f2a10038f20 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2a1804cff0 con 0x7f2a281084d0 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a277fe700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:23.855+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:24.056+0000 7f2a277fe700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:24.056+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:24.457+0000 7f2a277fe700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:24.457+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:25.258+0000 7f2a277fe700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:25.258+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:28:29.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:26.859+0000 7f2a277fe700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2044834522 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:26.859+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:28.347+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f2a18003a50 con 0x7f2a281084d0 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:28.347+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 msgr2=0x7f2a1003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:28.347+0000 7f2a257fa700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.348+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f2a1804eb70 con 0x7f2a281084d0 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.348+0000 7f2a257fa700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 0x7f2a1003ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.348+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a1800af30 con 0x7f2a1003c810 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.348+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 0x7f2a1003ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.349+0000 7f2a277fe700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 0x7f2a1003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c003a10 tx=0x7f2a1c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.350+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f2a1800af30 con 0x7f2a1003c810 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.356+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2a28069250 con 0x7f2a1003c810 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a257fa700 1 -- 172.21.15.145:0/556404804 <== mgr.14120 v2:172.21.15.145:6800/2089479814 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2a28069250 con 0x7f2a1003c810 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 msgr2=0x7f2a1003ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:29.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 0x7f2a1003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c003a10 tx=0x7f2a1c0092b0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 msgr2=0x7f2a28107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009fb0 tx=0x7f2a18011bf0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 shutdown_connections 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f2a1003c810 0x7f2a1003ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a281084d0 0x7f2a28107d50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 --2- 172.21.15.145:0/556404804 >> [v2:172.21.15.145:6800/2044834522,v1:172.21.15.145:6801/2044834522] conn(0x7f2a10038f20 0x7f2a1003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 >> 172.21.15.145:0/556404804 conn(0x7f2a28101f20 msgr2=0x7f2a28102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 shutdown_connections 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:29.357+0000 7f2a2e529700 1 -- 172.21.15.145:0/556404804 wait complete. 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:mgr epoch 5 is available 2024-03-28T18:28:29.919 INFO:teuthology.orchestra.run.smithi145.stdout:Setting orchestrator backend to cephadm... 2024-03-28T18:28:30.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:30 smithi145 ceph-mon[32151]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T18:28:30.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:30 smithi145 ceph-mon[32151]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T18:28:30.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:30 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:31.031 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.330+0000 7f5edb7d2700 1 Processor -- start 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5edb7d2700 1 -- start start 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5edb7d2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5edb7d2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed4108470 con 0x7f5ed4107a80 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ed956e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ed956e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58150/0 (socket says 172.21.15.145:58150) 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ed956e700 1 -- 172.21.15.145:0/2011509662 learned_addr learned my addr 172.21.15.145:0/2011509662 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ed956e700 1 -- 172.21.15.145:0/2011509662 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4108c80 con 0x7f5ed4107a80 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ed956e700 1 --2- 172.21.15.145:0/2011509662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00098a0 tx=0x7f5ed0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439e5e0515e50867 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ecbfff700 1 -- 172.21.15.145:0/2011509662 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed0004160 con 0x7f5ed4107a80 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.331+0000 7f5ecbfff700 1 -- 172.21.15.145:0/2011509662 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ed0004300 con 0x7f5ed4107a80 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5ecbfff700 1 -- 172.21.15.145:0/2011509662 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed0011b50 con 0x7f5ed4107a80 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 -- 172.21.15.145:0/2011509662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 msgr2=0x7f5ed4107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/2011509662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00098a0 tx=0x7f5ed0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 -- 172.21.15.145:0/2011509662 shutdown_connections 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/2011509662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed4107ea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 -- 172.21.15.145:0/2011509662 >> 172.21.15.145:0/2011509662 conn(0x7f5ed41030c0 msgr2=0x7f5ed41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 -- 172.21.15.145:0/2011509662 shutdown_connections 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.332+0000 7f5edb7d2700 1 -- 172.21.15.145:0/2011509662 wait complete. 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5edb7d2700 1 Processor -- start 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5edb7d2700 1 -- start start 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5edb7d2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5edb7d2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed410c030 con 0x7f5ed4107a80 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5ed956e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:31.037 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5ed956e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58164/0 (socket says 172.21.15.145:58164) 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5ed956e700 1 -- 172.21.15.145:0/4267300024 learned_addr learned my addr 172.21.15.145:0/4267300024 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5ed956e700 1 -- 172.21.15.145:0/4267300024 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed419bf90 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.333+0000 7f5ed956e700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0011670 tx=0x7f5ed0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00039c0 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ed0011e50 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed419c220 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed001a5c0 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed419ef10 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f5ed001a8c0 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.335+0000 7f5eca7fc700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 0x7f5ec003b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.334+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed40637c0 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.335+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5ed004d040 con 0x7f5ed4107a80 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.335+0000 7f5ed8d6d700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 0x7f5ec003b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:31.038 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.335+0000 7f5ed8d6d700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 0x7f5ec003b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4006fd0 tx=0x7f5ec4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.340+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ed0014e20 con 0x7f5ed4107a80 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.401+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5ed002d910 con 0x7f5ed4107a80 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.556+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ed4104340 con 0x7f5ec0038dc0 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.563+0000 7f5eca7fc700 1 -- 172.21.15.145:0/4267300024 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5ed4104340 con 0x7f5ec0038dc0 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 msgr2=0x7f5ec003b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 0x7f5ec003b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4006fd0 tx=0x7f5ec4006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 msgr2=0x7f5ed419ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0011670 tx=0x7f5ed0011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 shutdown_connections 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f5ec0038dc0 0x7f5ec003b280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 --2- 172.21.15.145:0/4267300024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ed4107a80 0x7f5ed419ba50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 >> 172.21.15.145:0/4267300024 conn(0x7f5ed41030c0 msgr2=0x7f5ed4103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 shutdown_connections 2024-03-28T18:28:31.039 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:30.566+0000 7f5edb7d2700 1 -- 172.21.15.145:0/4267300024 wait complete. 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:30] ENGINE Bus STARTING 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:30] ENGINE Serving on https://172.21.15.145:7150 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:30] ENGINE Serving on http://172.21.15.145:8765 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:30] ENGINE Bus STARTED 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: mgrmap e8: x(active, since 2s) 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:31.644 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:31 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:32.225 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T18:28:32.225 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.438+0000 7fda1c409700 1 Processor -- start 2024-03-28T18:28:32.225 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.438+0000 7fda1c409700 1 -- start start 2024-03-28T18:28:32.225 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.438+0000 7fda1c409700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.438+0000 7fda1c409700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1407c910 con 0x7fda1407dee0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.438+0000 7fda1a1a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda1a1a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58176/0 (socket says 172.21.15.145:58176) 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda1a1a5700 1 -- 172.21.15.145:0/3115265602 learned_addr learned my addr 172.21.15.145:0/3115265602 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda1a1a5700 1 -- 172.21.15.145:0/3115265602 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1407ca90 con 0x7fda1407dee0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda1a1a5700 1 --2- 172.21.15.145:0/3115265602 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda10009e80 tx=0x7fda1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185087a93cb34e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda191a3700 1 -- 172.21.15.145:0/3115265602 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda10017070 con 0x7fda1407dee0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda191a3700 1 -- 172.21.15.145:0/3115265602 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda1000b8c0 con 0x7fda1407dee0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.439+0000 7fda191a3700 1 -- 172.21.15.145:0/3115265602 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda100045c0 con 0x7fda1407dee0 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- 172.21.15.145:0/3115265602 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 msgr2=0x7fda1407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 --2- 172.21.15.145:0/3115265602 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fda10009e80 tx=0x7fda1000b330 comp rx=0 tx=0).stop 2024-03-28T18:28:32.226 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- 172.21.15.145:0/3115265602 shutdown_connections 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 --2- 172.21.15.145:0/3115265602 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1407c340 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- 172.21.15.145:0/3115265602 >> 172.21.15.145:0/3115265602 conn(0x7fda14103020 msgr2=0x7fda14105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- 172.21.15.145:0/3115265602 shutdown_connections 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- 172.21.15.145:0/3115265602 wait complete. 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 Processor -- start 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- start start 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.440+0000 7fda1c409700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1410e070 con 0x7fda1407dee0 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1a1a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1a1a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58180/0 (socket says 172.21.15.145:58180) 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1a1a5700 1 -- 172.21.15.145:0/1407671632 learned_addr learned my addr 172.21.15.145:0/1407671632 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1a1a5700 1 -- 172.21.15.145:0/1407671632 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1419e170 con 0x7fda1407dee0 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1a1a5700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda10009e50 tx=0x7fda1000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda10020020 con 0x7fda1407dee0 2024-03-28T18:28:32.227 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda10003b50 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1419e400 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda10017040 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.441+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda141a10f0 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.442+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fda10023020 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.442+0000 7fda0b7fe700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 0x7fda0003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.442+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fda1000f050 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.442+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda14069250 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.442+0000 7fda199a4700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 0x7fda0003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.443+0000 7fda199a4700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 0x7fda0003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda04006fd0 tx=0x7fda04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.447+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda1002a020 con 0x7fda1407dee0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.643+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fda14104270 con 0x7fda00038ed0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.643+0000 7fda0b7fe700 1 -- 172.21.15.145:0/1407671632 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fda14104270 con 0x7fda00038ed0 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.645+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 msgr2=0x7fda0003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:32.228 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.645+0000 7fda1c409700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 0x7fda0003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda04006fd0 tx=0x7fda04006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.645+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 msgr2=0x7fda1419dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.645+0000 7fda1c409700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fda10009e50 tx=0x7fda1000bd30 comp rx=0 tx=0).stop 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 shutdown_connections 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fda00038ed0 0x7fda0003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 --2- 172.21.15.145:0/1407671632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda1407dee0 0x7fda1419dc30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 >> 172.21.15.145:0/1407671632 conn(0x7fda14103020 msgr2=0x7fda14103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 shutdown_connections 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:31.646+0000 7fda1c409700 1 -- 172.21.15.145:0/1407671632 wait complete. 2024-03-28T18:28:32.229 INFO:teuthology.orchestra.run.smithi145.stdout:Generating ssh key... 2024-03-28T18:28:32.875 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:32 smithi145 ceph-mon[32151]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:33.442 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: Generating public/private rsa key pair. 2024-03-28T18:28:33.442 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: Your identification has been saved in /tmp/tmpef91kiqw/key. 2024-03-28T18:28:33.442 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: Your public key has been saved in /tmp/tmpef91kiqw/key.pub. 2024-03-28T18:28:33.442 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: The key fingerprint is: 2024-03-28T18:28:33.442 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: SHA256:/vn1SMrxhR0nVNbVd3flQsa7E1wLwznLkFhQyzRwGP4 ceph-b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: The key's randomart image is: 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: +---[RSA 3072]----+ 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | +O*o.+ X| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | .o+ooO.+O| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | . oo.BoB| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | . ++o | 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | S E .+.| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | . o+o| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | . . +.o| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | . o * + | 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: | o.+ o .| 2024-03-28T18:28:33.443 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: +----[SHA256]-----+ 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.668+0000 7f9df7a07700 1 Processor -- start 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.668+0000 7f9df7a07700 1 -- start start 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.668+0000 7f9df7a07700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.668+0000 7f9df7a07700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df007c910 con 0x7f9df007de90 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9df57a3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9df57a3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38104/0 (socket says 172.21.15.145:38104) 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9df57a3700 1 -- 172.21.15.145:0/657398212 learned_addr learned my addr 172.21.15.145:0/657398212 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9df57a3700 1 -- 172.21.15.145:0/657398212 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df007ca90 con 0x7f9df007de90 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9df57a3700 1 --2- 172.21.15.145:0/657398212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0098a0 tx=0x7f9dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a38ee06b642bf93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9de7fff700 1 -- 172.21.15.145:0/657398212 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dec004160 con 0x7f9df007de90 2024-03-28T18:28:33.733 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9de7fff700 1 -- 172.21.15.145:0/657398212 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dec004300 con 0x7f9df007de90 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.669+0000 7f9de7fff700 1 -- 172.21.15.145:0/657398212 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dec011b50 con 0x7f9df007de90 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 -- 172.21.15.145:0/657398212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 msgr2=0x7f9df007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 --2- 172.21.15.145:0/657398212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0098a0 tx=0x7f9dec009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 -- 172.21.15.145:0/657398212 shutdown_connections 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 --2- 172.21.15.145:0/657398212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df007c340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 -- 172.21.15.145:0/657398212 >> 172.21.15.145:0/657398212 conn(0x7f9df0103130 msgr2=0x7f9df0105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 -- 172.21.15.145:0/657398212 shutdown_connections 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.670+0000 7f9df7a07700 1 -- 172.21.15.145:0/657398212 wait complete. 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df7a07700 1 Processor -- start 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df7a07700 1 -- start start 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df7a07700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df7a07700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df010e0e0 con 0x7f9df007de90 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df57a3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:33.734 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df57a3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38118/0 (socket says 172.21.15.145:38118) 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df57a3700 1 -- 172.21.15.145:0/2170401924 learned_addr learned my addr 172.21.15.145:0/2170401924 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df57a3700 1 -- 172.21.15.145:0/2170401924 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df019e200 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.671+0000 7f9df57a3700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9dec011b60 tx=0x7f9dec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.673+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dec003970 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.673+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dec003b10 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.673+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dec024610 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.673+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df019e490 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.673+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df01a1180 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.674+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9dec0248c0 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.674+0000 7f9de67fc700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 0x7f9ddc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.674+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0069250 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.674+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9dec011e00 con 0x7f9df007de90 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.674+0000 7f9df4fa2700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 0x7f9ddc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.675+0000 7f9df4fa2700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 0x7f9ddc03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006fd0 tx=0x7f9de0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.680+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9dec00ba00 con 0x7f9df007de90 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:32.874+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9df0104380 con 0x7f9ddc038f70 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.157+0000 7f9de67fc700 1 -- 172.21.15.145:0/2170401924 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9df0104380 con 0x7f9ddc038f70 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 msgr2=0x7f9ddc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 0x7f9ddc03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006fd0 tx=0x7f9de0006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 msgr2=0x7f9df019dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9dec011b60 tx=0x7f9dec011c40 comp rx=0 tx=0).stop 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 shutdown_connections 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9ddc038f70 0x7f9ddc03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 --2- 172.21.15.145:0/2170401924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df007de90 0x7f9df019dcc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 >> 172.21.15.145:0/2170401924 conn(0x7f9df0103130 msgr2=0x7f9df0103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 shutdown_connections 2024-03-28T18:28:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:33.160+0000 7f9df7a07700 1 -- 172.21.15.145:0/2170401924 wait complete. 2024-03-28T18:28:34.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:34 smithi145 ceph-mon[32151]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:34.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:34 smithi145 ceph-mon[32151]: Generating ssh key... 2024-03-28T18:28:34.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:34 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:34.341 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:34 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:34.914 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.127+0000 7efe34bb0700 1 Processor -- start 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.127+0000 7efe34bb0700 1 -- start start 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.127+0000 7efe34bb0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.127+0000 7efe34bb0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3007c910 con 0x7efe3007dee0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38122/0 (socket says 172.21.15.145:38122) 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2e59c700 1 -- 172.21.15.145:0/2198718021 learned_addr learned my addr 172.21.15.145:0/2198718021 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2e59c700 1 -- 172.21.15.145:0/2198718021 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3007ca90 con 0x7efe3007dee0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2e59c700 1 --2- 172.21.15.145:0/2198718021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe200098a0 tx=0x7efe20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa021472d51494d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2d59a700 1 -- 172.21.15.145:0/2198718021 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20004160 con 0x7efe3007dee0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2d59a700 1 -- 172.21.15.145:0/2198718021 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe20004300 con 0x7efe3007dee0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.128+0000 7efe2d59a700 1 -- 172.21.15.145:0/2198718021 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20011b50 con 0x7efe3007dee0 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 -- 172.21.15.145:0/2198718021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 msgr2=0x7efe3007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:34.915 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 --2- 172.21.15.145:0/2198718021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe200098a0 tx=0x7efe20009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 -- 172.21.15.145:0/2198718021 shutdown_connections 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 --2- 172.21.15.145:0/2198718021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3007c340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 -- 172.21.15.145:0/2198718021 >> 172.21.15.145:0/2198718021 conn(0x7efe30103050 msgr2=0x7efe30105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 -- 172.21.15.145:0/2198718021 shutdown_connections 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.129+0000 7efe34bb0700 1 -- 172.21.15.145:0/2198718021 wait complete. 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe34bb0700 1 Processor -- start 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe34bb0700 1 -- start start 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe34bb0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe34bb0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3010e080 con 0x7efe3007dee0 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe2e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe2e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38132/0 (socket says 172.21.15.145:38132) 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.130+0000 7efe2e59c700 1 -- 172.21.15.145:0/1784531848 learned_addr learned my addr 172.21.15.145:0/1784531848 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe2e59c700 1 -- 172.21.15.145:0/1784531848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3019e1f0 con 0x7efe3007dee0 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe2e59c700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe20011850 tx=0x7efe20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:34.916 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20003af0 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe20020070 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3019e480 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe200195a0 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.131+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe301a1170 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.132+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efe20019850 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.132+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe30069250 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.132+0000 7efe1f7fe700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 0x7efe1803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.132+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe20011e00 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.132+0000 7efe2dd9b700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 0x7efe1803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.133+0000 7efe2dd9b700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 0x7efe1803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe24006fd0 tx=0x7efe24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.137+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efe20003c90 con 0x7efe3007dee0 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.340+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efe30104290 con 0x7efe18038f70 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.341+0000 7efe1f7fe700 1 -- 172.21.15.145:0/1784531848 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7efe30104290 con 0x7efe18038f70 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.343+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 msgr2=0x7efe1803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:34.917 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.343+0000 7efe34bb0700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 0x7efe1803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efe24006fd0 tx=0x7efe24006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.343+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 msgr2=0x7efe3019dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.343+0000 7efe34bb0700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe20011850 tx=0x7efe20011c10 comp rx=0 tx=0).stop 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 shutdown_connections 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7efe18038f70 0x7efe1803b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 --2- 172.21.15.145:0/1784531848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe3007dee0 0x7efe3019dcb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 >> 172.21.15.145:0/1784531848 conn(0x7efe30103050 msgr2=0x7efe30103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 shutdown_connections 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:34.344+0000 7efe34bb0700 1 -- 172.21.15.145:0/1784531848 wait complete. 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T18:28:34.918 INFO:teuthology.orchestra.run.smithi145.stdout:Adding host smithi145... 2024-03-28T18:28:35.226 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:35 smithi145 ceph-mon[32151]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:36.428 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:36 smithi145 ceph-mon[32151]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "addr": "172.21.15.145", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:37.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:37 smithi145 ceph-mon[32151]: Deploying cephadm binary to smithi145 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout Added host 'smithi145' with addr '172.21.15.145' 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.327+0000 7f7b473dc700 1 Processor -- start 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.327+0000 7f7b473dc700 1 -- start start 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.327+0000 7f7b473dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.327+0000 7f7b473dc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4007bed0 con 0x7f7b401084f0 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b45178700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b45178700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38140/0 (socket says 172.21.15.145:38140) 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b45178700 1 -- 172.21.15.145:0/3655694575 learned_addr learned my addr 172.21.15.145:0/3655694575 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b45178700 1 -- 172.21.15.145:0/3655694575 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4010ae20 con 0x7f7b401084f0 2024-03-28T18:28:40.280 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b45178700 1 --2- 172.21.15.145:0/3655694575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0098a0 tx=0x7f7b3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562b14df7423c617 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b37fff700 1 -- 172.21.15.145:0/3655694575 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b3c004160 con 0x7f7b401084f0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b37fff700 1 -- 172.21.15.145:0/3655694575 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b3c004300 con 0x7f7b401084f0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.328+0000 7f7b37fff700 1 -- 172.21.15.145:0/3655694575 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b3c011b50 con 0x7f7b401084f0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 -- 172.21.15.145:0/3655694575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 msgr2=0x7f7b4010a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 --2- 172.21.15.145:0/3655694575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0098a0 tx=0x7f7b3c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 -- 172.21.15.145:0/3655694575 shutdown_connections 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 --2- 172.21.15.145:0/3655694575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4010a8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 -- 172.21.15.145:0/3655694575 >> 172.21.15.145:0/3655694575 conn(0x7f7b40101f60 msgr2=0x7f7b40104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 -- 172.21.15.145:0/3655694575 shutdown_connections 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.329+0000 7f7b473dc700 1 -- 172.21.15.145:0/3655694575 wait complete. 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b473dc700 1 Processor -- start 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b473dc700 1 -- start start 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b473dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b473dc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4010be50 con 0x7f7b401084f0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b45178700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:40.281 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b45178700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38142/0 (socket says 172.21.15.145:38142) 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b45178700 1 -- 172.21.15.145:0/1933321133 learned_addr learned my addr 172.21.15.145:0/1933321133 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b45178700 1 -- 172.21.15.145:0/1933321133 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4019bf50 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b45178700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c011670 tx=0x7f7b3c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b3c0039c0 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b3c011e50 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b3c01a5c0 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.330+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4019c1e0 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.331+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4019eed0 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.331+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7b3c01a870 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.331+0000 7f7b367fc700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 0x7f7b2c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.331+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7b3c04d200 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.331+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400637c0 con 0x7f7b401084f0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.332+0000 7f7b44977700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 0x7f7b2c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.332+0000 7f7b44977700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 0x7f7b2c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:40.282 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.336+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b3c019c70 con 0x7f7b401084f0 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:35.535+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi145", "addr": "172.21.15.145", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b401031b0 con 0x7f7b2c038f20 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.752+0000 7f7b367fc700 1 -- 172.21.15.145:0/1933321133 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7b401031b0 con 0x7f7b2c038f20 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 msgr2=0x7f7b2c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 0x7f7b2c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b30006fd0 tx=0x7f7b30006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 msgr2=0x7f7b4019ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c011670 tx=0x7f7b3c011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 shutdown_connections 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f7b2c038f20 0x7f7b2c03b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 --2- 172.21.15.145:0/1933321133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b401084f0 0x7f7b4019ba10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 >> 172.21.15.145:0/1933321133 conn(0x7f7b40101f60 msgr2=0x7f7b40102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 shutdown_connections 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:39.755+0000 7f7b473dc700 1 -- 172.21.15.145:0/1933321133 wait complete. 2024-03-28T18:28:40.283 INFO:teuthology.orchestra.run.smithi145.stdout:Deploying unmanaged mon service... 2024-03-28T18:28:40.939 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:40 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:40.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:40 smithi145 ceph-mon[32151]: Added host smithi145 2024-03-28T18:28:40.940 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:40 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad2d1c700 1 Processor -- start 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad2d1c700 1 -- start start 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad2d1c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad2d1c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc07be90 con 0x7f9acc108580 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad0ab8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad0ab8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38150/0 (socket says 172.21.15.145:38150) 2024-03-28T18:28:41.428 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.709+0000 7f9ad0ab8700 1 -- 172.21.15.145:0/3786539250 learned_addr learned my addr 172.21.15.145:0/3786539250 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.710+0000 7f9ad0ab8700 1 -- 172.21.15.145:0/3786539250 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc10aeb0 con 0x7f9acc108580 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.710+0000 7f9ad0ab8700 1 --2- 172.21.15.145:0/3786539250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00098a0 tx=0x7f9ac0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1191fd1548b57c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.710+0000 7f9acb7fe700 1 -- 172.21.15.145:0/3786539250 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ac0004160 con 0x7f9acc108580 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.710+0000 7f9acb7fe700 1 -- 172.21.15.145:0/3786539250 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ac0004300 con 0x7f9acc108580 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.710+0000 7f9acb7fe700 1 -- 172.21.15.145:0/3786539250 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ac0011b50 con 0x7f9acc108580 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3786539250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 msgr2=0x7f9acc10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3786539250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00098a0 tx=0x7f9ac0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3786539250 shutdown_connections 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3786539250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc10a970 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3786539250 >> 172.21.15.145:0/3786539250 conn(0x7f9acc101f90 msgr2=0x7f9acc1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3786539250 shutdown_connections 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3786539250 wait complete. 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 Processor -- start 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.711+0000 7f9ad2d1c700 1 -- start start 2024-03-28T18:28:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad2d1c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad2d1c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc10bed0 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad0ab8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad0ab8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38156/0 (socket says 172.21.15.145:38156) 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad0ab8700 1 -- 172.21.15.145:0/3626522050 learned_addr learned my addr 172.21.15.145:0/3626522050 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad0ab8700 1 -- 172.21.15.145:0/3626522050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc19bf90 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad0ab8700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0011670 tx=0x7f9ac0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ac00039c0 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ac0011e50 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc19c220 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ac001a5c0 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.712+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc19ef10 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.713+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9ac001a870 con 0x7f9acc108580 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.713+0000 7f9ac9ffb700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 0x7f9abc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:41.430 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.713+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9ac004d320 con 0x7f9acc108580 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.713+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc0637c0 con 0x7f9acc108580 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.714+0000 7f9acbfff700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 0x7f9abc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.714+0000 7f9acbfff700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 0x7f9abc03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.718+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9ac002cc80 con 0x7f9acc108580 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.939+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9acc1031b0 con 0x7f9abc038f20 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.947+0000 7f9ac9ffb700 1 -- 172.21.15.145:0/3626522050 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9acc1031b0 con 0x7f9abc038f20 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 msgr2=0x7f9abc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 0x7f9abc03b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8006fd0 tx=0x7f9ab8006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 msgr2=0x7f9acc19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0011670 tx=0x7f9ac0011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 shutdown_connections 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f9abc038f20 0x7f9abc03b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 --2- 172.21.15.145:0/3626522050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9acc108580 0x7f9acc19ba50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 >> 172.21.15.145:0/3626522050 conn(0x7f9acc101f90 msgr2=0x7f9acc102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 shutdown_connections 2024-03-28T18:28:41.431 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:40.949+0000 7f9ad2d1c700 1 -- 172.21.15.145:0/3626522050 wait complete. 2024-03-28T18:28:41.432 INFO:teuthology.orchestra.run.smithi145.stdout:Deploying unmanaged mgr service... 2024-03-28T18:28:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:41 smithi145 ceph-mon[32151]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:41 smithi145 ceph-mon[32151]: Saving service mon spec with placement count:5 2024-03-28T18:28:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:41 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:42.618 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T18:28:42.618 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f390ad34700 1 Processor -- start 2024-03-28T18:28:42.618 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f390ad34700 1 -- start start 2024-03-28T18:28:42.618 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f390ad34700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:42.618 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f390ad34700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041073c0 con 0x7f39041069d0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f3908ad0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f3908ad0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38164/0 (socket says 172.21.15.145:38164) 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f3908ad0700 1 -- 172.21.15.145:0/1784504261 learned_addr learned my addr 172.21.15.145:0/1784504261 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.848+0000 7f3908ad0700 1 -- 172.21.15.145:0/1784504261 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904107bd0 con 0x7f39041069d0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f3908ad0700 1 --2- 172.21.15.145:0/1784504261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38f80098a0 tx=0x7f38f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c7ac7c8cfdf8b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f39037fe700 1 -- 172.21.15.145:0/1784504261 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38f8004160 con 0x7f39041069d0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f39037fe700 1 -- 172.21.15.145:0/1784504261 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38f8004300 con 0x7f39041069d0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f39037fe700 1 -- 172.21.15.145:0/1784504261 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38f8011b50 con 0x7f39041069d0 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 -- 172.21.15.145:0/1784504261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 msgr2=0x7f3904106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 --2- 172.21.15.145:0/1784504261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38f80098a0 tx=0x7f38f8009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 -- 172.21.15.145:0/1784504261 shutdown_connections 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 --2- 172.21.15.145:0/1784504261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f3904106df0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 -- 172.21.15.145:0/1784504261 >> 172.21.15.145:0/1784504261 conn(0x7f3904101f90 msgr2=0x7f39041043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 -- 172.21.15.145:0/1784504261 shutdown_connections 2024-03-28T18:28:42.619 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.849+0000 7f390ad34700 1 -- 172.21.15.145:0/1784504261 wait complete. 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f390ad34700 1 Processor -- start 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f390ad34700 1 -- start start 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f390ad34700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f390ad34700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390410af80 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f3908ad0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f3908ad0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38172/0 (socket says 172.21.15.145:38172) 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f3908ad0700 1 -- 172.21.15.145:0/2052011292 learned_addr learned my addr 172.21.15.145:0/2052011292 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.850+0000 7f3908ad0700 1 -- 172.21.15.145:0/2052011292 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390419c080 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f3908ad0700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38f8011670 tx=0x7f38f8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38f80039c0 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38f8011e50 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38f801a5c0 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390419c310 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.851+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390419f000 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.852+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f38f8024430 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.852+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040637c0 con 0x7f39041069d0 2024-03-28T18:28:42.620 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.852+0000 7f3901ffb700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 0x7f38f403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.852+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f38f804d250 con 0x7f39041069d0 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.852+0000 7f3903fff700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 0x7f38f403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.853+0000 7f3903fff700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 0x7f38f403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006fd0 tx=0x7f38f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:41.857+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f38f8003b60 con 0x7f39041069d0 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.067+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f39040008d0 con 0x7f38f4038f20 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.074+0000 7f3901ffb700 1 -- 172.21.15.145:0/2052011292 <== mgr.14120 v2:172.21.15.145:6800/2089479814 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f39040008d0 con 0x7f38f4038f20 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 msgr2=0x7f38f403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 0x7f38f403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006fd0 tx=0x7f38f0006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 msgr2=0x7f390419bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38f8011670 tx=0x7f38f8011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 shutdown_connections 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f38f4038f20 0x7f38f403b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 --2- 172.21.15.145:0/2052011292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39041069d0 0x7f390419bb40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 >> 172.21.15.145:0/2052011292 conn(0x7f3904101f90 msgr2=0x7f3904102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:42.621 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 shutdown_connections 2024-03-28T18:28:42.622 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:42.076+0000 7f390ad34700 1 -- 172.21.15.145:0/2052011292 wait complete. 2024-03-28T18:28:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:43 smithi145 ceph-mon[32151]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:43 smithi145 ceph-mon[32151]: Saving service mgr spec with placement count:2 2024-03-28T18:28:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:43 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:43.824 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.042+0000 7fb754642700 1 Processor -- start 2024-03-28T18:28:43.824 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.042+0000 7fb754642700 1 -- start start 2024-03-28T18:28:43.824 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb754642700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb754642700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c07be90 con 0x7fb74c108560 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7523de700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7523de700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60314/0 (socket says 172.21.15.145:60314) 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7523de700 1 -- 172.21.15.145:0/122811009 learned_addr learned my addr 172.21.15.145:0/122811009 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7523de700 1 -- 172.21.15.145:0/122811009 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c10ae90 con 0x7fb74c108560 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7523de700 1 --2- 172.21.15.145:0/122811009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a21536f3b6b2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7513dc700 1 -- 172.21.15.145:0/122811009 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb73c004160 con 0x7fb74c108560 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7513dc700 1 -- 172.21.15.145:0/122811009 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb73c004300 con 0x7fb74c108560 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.043+0000 7fb7513dc700 1 -- 172.21.15.145:0/122811009 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb73c011b50 con 0x7fb74c108560 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 -- 172.21.15.145:0/122811009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 msgr2=0x7fb74c10a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 --2- 172.21.15.145:0/122811009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 -- 172.21.15.145:0/122811009 shutdown_connections 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 --2- 172.21.15.145:0/122811009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c10a950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 -- 172.21.15.145:0/122811009 >> 172.21.15.145:0/122811009 conn(0x7fb74c101f50 msgr2=0x7fb74c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:43.825 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 -- 172.21.15.145:0/122811009 shutdown_connections 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.044+0000 7fb754642700 1 -- 172.21.15.145:0/122811009 wait complete. 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 Processor -- start 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 -- start start 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c10beb0 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7523de700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7523de700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60322/0 (socket says 172.21.15.145:60322) 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7523de700 1 -- 172.21.15.145:0/134769565 learned_addr learned my addr 172.21.15.145:0/134769565 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7523de700 1 -- 172.21.15.145:0/134769565 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c19bff0 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7523de700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fb0 tx=0x7fb73c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb73c0038b0 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb73c011e50 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c19c280 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb73c019420 con 0x7fb74c108560 2024-03-28T18:28:43.826 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.045+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c19ef70 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.046+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb73c019720 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.046+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c069250 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.046+0000 7fb7437fe700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 0x7fb73803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.046+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb73c04d240 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.047+0000 7fb751bdd700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 0x7fb73803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.047+0000 7fb751bdd700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 0x7fb73803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb748006fd0 tx=0x7fb748006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.051+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb73c01a430 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.243+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb74c19f5c0 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.248+0000 7fb7437fe700 1 -- 172.21.15.145:0/134769565 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb73c019a10 con 0x7fb74c108560 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.250+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 msgr2=0x7fb73803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.250+0000 7fb754642700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 0x7fb73803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb748006fd0 tx=0x7fb748006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.250+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 msgr2=0x7fb74c19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.250+0000 7fb754642700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fb0 tx=0x7fb73c011bf0 comp rx=0 tx=0).stop 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 shutdown_connections 2024-03-28T18:28:43.827 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fb738038ed0 0x7fb73803b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:43.828 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 --2- 172.21.15.145:0/134769565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb74c108560 0x7fb74c19bab0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:43.828 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 >> 172.21.15.145:0/134769565 conn(0x7fb74c101f50 msgr2=0x7fb74c102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:43.828 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 shutdown_connections 2024-03-28T18:28:43.828 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:43.251+0000 7fb754642700 1 -- 172.21.15.145:0/134769565 wait complete. 2024-03-28T18:28:44.340 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:44 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/134769565' entity='client.admin' 2024-03-28T18:28:45.536 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.306+0000 7f96f3fe0700 1 Processor -- start 2024-03-28T18:28:45.536 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.306+0000 7f96f3fe0700 1 -- start start 2024-03-28T18:28:45.536 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f3fe0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:45.536 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f3fe0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec107370 con 0x7f96ec106980 2024-03-28T18:28:45.536 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f1d7c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f1d7c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60324/0 (socket says 172.21.15.145:60324) 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f1d7c700 1 -- 172.21.15.145:0/1432848014 learned_addr learned my addr 172.21.15.145:0/1432848014 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f1d7c700 1 -- 172.21.15.145:0/1432848014 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec107b80 con 0x7f96ec106980 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f1d7c700 1 --2- 172.21.15.145:0/1432848014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9200909e7e68c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f0d7a700 1 -- 172.21.15.145:0/1432848014 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e8004160 con 0x7f96ec106980 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f0d7a700 1 -- 172.21.15.145:0/1432848014 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96e8004300 con 0x7f96ec106980 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.307+0000 7f96f0d7a700 1 -- 172.21.15.145:0/1432848014 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e8011b50 con 0x7f96ec106980 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 -- 172.21.15.145:0/1432848014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 msgr2=0x7f96ec106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/1432848014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e8009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 -- 172.21.15.145:0/1432848014 shutdown_connections 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/1432848014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec106da0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 -- 172.21.15.145:0/1432848014 >> 172.21.15.145:0/1432848014 conn(0x7f96ec101f60 msgr2=0x7f96ec104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 -- 172.21.15.145:0/1432848014 shutdown_connections 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.308+0000 7f96f3fe0700 1 -- 172.21.15.145:0/1432848014 wait complete. 2024-03-28T18:28:45.537 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f3fe0700 1 Processor -- start 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f3fe0700 1 -- start start 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f3fe0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f3fe0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec10af30 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f1d7c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f1d7c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60332/0 (socket says 172.21.15.145:60332) 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f1d7c700 1 -- 172.21.15.145:0/17539603 learned_addr learned my addr 172.21.15.145:0/17539603 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f1d7c700 1 -- 172.21.15.145:0/17539603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec192070 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f1d7c700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96e8011670 tx=0x7f96e8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e80039c0 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96e8011e50 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec190260 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.309+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96e801a5c0 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.310+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec1907a0 con 0x7f96ec106980 2024-03-28T18:28:45.538 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.310+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f96e801a8c0 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.310+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec0637c0 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.310+0000 7f96e2ffd700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 0x7f96d803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.311+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f96e804d430 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.311+0000 7f96f157b700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 0x7f96d803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.311+0000 7f96f157b700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 0x7f96d803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96dc006fd0 tx=0x7f96dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.315+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96e8019c70 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.540+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f96ec195920 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.542+0000 7f96e2ffd700 1 -- 172.21.15.145:0/17539603 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) v1 ==== 130+0+0 (secure 0 0 0) 0x7f96e802cbc0 con 0x7f96ec106980 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.544+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 msgr2=0x7f96d803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.544+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 0x7f96d803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f96dc006fd0 tx=0x7f96dc006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.544+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 msgr2=0x7f96ec191b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.544+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96e8011670 tx=0x7f96e8011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 shutdown_connections 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f96d8038ed0 0x7f96d803b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:45.539 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 --2- 172.21.15.145:0/17539603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96ec106980 0x7f96ec191b30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:45.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 >> 172.21.15.145:0/17539603 conn(0x7f96ec101f60 msgr2=0x7f96ec102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:45.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 shutdown_connections 2024-03-28T18:28:45.540 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:44.545+0000 7f96f3fe0700 1 -- 172.21.15.145:0/17539603 wait complete. 2024-03-28T18:28:45.540 INFO:teuthology.orchestra.run.smithi145.stdout:Enabling the dashboard module... 2024-03-28T18:28:45.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:45 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/17539603' entity='client.admin' 2024-03-28T18:28:46.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:46 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3235711688' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T18:28:46.879 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:46 smithi145 ceph-mon[32151]: from='mgr.14120 172.21.15.145:0/3226195784' entity='mgr.x' 2024-03-28T18:28:46.880 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: ignoring --setuser ceph since I am not root 2024-03-28T18:28:46.880 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: ignoring --setgroup ceph since I am not root 2024-03-28T18:28:47.210 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:47.117+0000 7f0fc9a98200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:28:47.210 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:47.209+0000 7f0fc9a98200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:28:47.220 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7c4eb2700 1 Processor -- start 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7c4eb2700 1 -- start start 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7c4eb2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7c4eb2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0109aa0 con 0x7fa7c01090b0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7be59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7be59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60338/0 (socket says 172.21.15.145:60338) 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.907+0000 7fa7be59c700 1 -- 172.21.15.145:0/3958744208 learned_addr learned my addr 172.21.15.145:0/3958744208 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.908+0000 7fa7be59c700 1 -- 172.21.15.145:0/3958744208 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0109c20 con 0x7fa7c01090b0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.908+0000 7fa7be59c700 1 --2- 172.21.15.145:0/3958744208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00098a0 tx=0x7fa7b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe322f9929190e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.908+0000 7fa7bd59a700 1 -- 172.21.15.145:0/3958744208 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7b0004160 con 0x7fa7c01090b0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.908+0000 7fa7bd59a700 1 -- 172.21.15.145:0/3958744208 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7b0004300 con 0x7fa7c01090b0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.908+0000 7fa7bd59a700 1 -- 172.21.15.145:0/3958744208 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7b0011b50 con 0x7fa7c01090b0 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3958744208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 msgr2=0x7fa7c01094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3958744208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00098a0 tx=0x7fa7b0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:47.221 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3958744208 shutdown_connections 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3958744208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01094d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3958744208 >> 172.21.15.145:0/3958744208 conn(0x7fa7c01016d0 msgr2=0x7fa7c0103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3958744208 shutdown_connections 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.909+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3958744208 wait complete. 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7c4eb2700 1 Processor -- start 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7c4eb2700 1 -- start start 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7c4eb2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7c4eb2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0106920 con 0x7fa7c01090b0 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7be59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7be59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60350/0 (socket says 172.21.15.145:60350) 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7be59c700 1 -- 172.21.15.145:0/3235711688 learned_addr learned my addr 172.21.15.145:0/3235711688 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7be59c700 1 -- 172.21.15.145:0/3235711688 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0108c10 con 0x7fa7c01090b0 2024-03-28T18:28:47.222 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7be59c700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0011670 tx=0x7fa7b0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.910+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7b00039c0 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.911+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7b0011e50 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.911+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c0106e00 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.911+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7b001a5c0 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.911+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c0107340 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa7b001a8c0 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c00637c0 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7af7fe700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 0x7fa7a803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa7b004d220 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7bdd9b700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 0x7fa7a803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.912+0000 7fa7bdd9b700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 0x7fa7a803b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4006fd0 tx=0x7fa7b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:45.917+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa7b0019c70 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.172+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa7c0107620 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.371+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 7 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b0018dc0 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.545+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa7b0020240 con 0x7fa7c01090b0 2024-03-28T18:28:47.223 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.547+0000 7fa7af7fe700 1 -- 172.21.15.145:0/3235711688 <== mon.0 v2:172.21.15.145:3300/0 9 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fa7b0018350 con 0x7fa7c01090b0 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 msgr2=0x7fa7a803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 0x7fa7a803b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4006fd0 tx=0x7fa7b4006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 msgr2=0x7fa7c01086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0011670 tx=0x7fa7b0011b60 comp rx=0 tx=0).stop 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 shutdown_connections 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7fa7a8038ed0 0x7fa7a803b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 --2- 172.21.15.145:0/3235711688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7c01090b0 0x7fa7c01086d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 >> 172.21.15.145:0/3235711688 conn(0x7fa7c01016d0 msgr2=0x7fa7c0102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 shutdown_connections 2024-03-28T18:28:47.224 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:46.549+0000 7fa7c4eb2700 1 -- 172.21.15.145:0/3235711688 wait complete. 2024-03-28T18:28:47.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3235711688' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T18:28:47.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-mon[32151]: mgrmap e9: x(active, since 18s) 2024-03-28T18:28:48.112 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:47.872+0000 7f0fc9a98200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:28:48.112 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:47.972+0000 7f0fc9a98200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:28:48.112 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.112+0000 7f0fc9a98200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:28:48.469 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.209+0000 7f0fc9a98200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:28:48.469 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.291+0000 7f0fc9a98200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:28:48.477 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:48.477 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T18:28:48.477 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:28:48.477 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.598+0000 7f3d8862d700 1 Processor -- start 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.598+0000 7f3d8862d700 1 -- start start 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d8862d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d8862d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8007c910 con 0x7f3d8007dee0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d863c9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d863c9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60370/0 (socket says 172.21.15.145:60370) 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d863c9700 1 -- 172.21.15.145:0/2461083557 learned_addr learned my addr 172.21.15.145:0/2461083557 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d863c9700 1 -- 172.21.15.145:0/2461083557 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8007ca90 con 0x7f3d8007dee0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d863c9700 1 --2- 172.21.15.145:0/2461083557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0098a0 tx=0x7f3d7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bada7753d05afc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d853c7700 1 -- 172.21.15.145:0/2461083557 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d7c004160 con 0x7f3d8007dee0 2024-03-28T18:28:48.478 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.599+0000 7f3d853c7700 1 -- 172.21.15.145:0/2461083557 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d7c004300 con 0x7f3d8007dee0 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d853c7700 1 -- 172.21.15.145:0/2461083557 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d7c011b20 con 0x7f3d8007dee0 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 -- 172.21.15.145:0/2461083557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 msgr2=0x7f3d8007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 --2- 172.21.15.145:0/2461083557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0098a0 tx=0x7f3d7c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 -- 172.21.15.145:0/2461083557 shutdown_connections 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 --2- 172.21.15.145:0/2461083557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8007c340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 -- 172.21.15.145:0/2461083557 >> 172.21.15.145:0/2461083557 conn(0x7f3d80103020 msgr2=0x7f3d80105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 -- 172.21.15.145:0/2461083557 shutdown_connections 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.600+0000 7f3d8862d700 1 -- 172.21.15.145:0/2461083557 wait complete. 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d8862d700 1 Processor -- start 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d8862d700 1 -- start start 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d8862d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d8862d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8010e0c0 con 0x7f3d8007dee0 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d863c9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:48.479 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d863c9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60380/0 (socket says 172.21.15.145:60380) 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d863c9700 1 -- 172.21.15.145:0/3274209866 learned_addr learned my addr 172.21.15.145:0/3274209866 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d863c9700 1 -- 172.21.15.145:0/3274209866 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8019e190 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.601+0000 7f3d863c9700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0116a0 tx=0x7f3d7c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.602+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d7c003a40 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.602+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d7c003be0 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.602+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8019e420 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.602+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d7c01a720 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.602+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8019e880 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f3d7c01aa20 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d80069250 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d777fe700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 0x7f3d6c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3d7c04d4d0 con 0x7f3d8007dee0 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d85bc8700 1 -- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 msgr2=0x7f3d6c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.603+0000 7f3d85bc8700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 0x7f3d6c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:28:48.480 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.608+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d7c011e20 con 0x7f3d8007dee0 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.804+0000 7f3d85bc8700 1 -- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 msgr2=0x7f3d6c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.804+0000 7f3d85bc8700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 0x7f3d6c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.862+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3d801a12a0 con 0x7f3d8007dee0 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.863+0000 7f3d777fe700 1 -- 172.21.15.145:0/3274209866 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f3d7c024b50 con 0x7f3d8007dee0 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 msgr2=0x7f3d6c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 0x7f3d6c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 msgr2=0x7f3d8019dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0116a0 tx=0x7f3d7c011ba0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 shutdown_connections 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f3d6c038f20 0x7f3d6c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 --2- 172.21.15.145:0/3274209866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d8007dee0 0x7f3d8019dc50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 >> 172.21.15.145:0/3274209866 conn(0x7f3d80103020 msgr2=0x7f3d80103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 shutdown_connections 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:47.865+0000 7f3d8862d700 1 -- 172.21.15.145:0/3274209866 wait complete. 2024-03-28T18:28:48.481 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for the mgr to restart... 2024-03-28T18:28:48.482 INFO:teuthology.orchestra.run.smithi145.stdout:Waiting for mgr epoch 9... 2024-03-28T18:28:48.763 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3274209866' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T18:28:48.763 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.562+0000 7f0fc9a98200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:28:48.764 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.675+0000 7f0fc9a98200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:48.764 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.763+0000 7f0fc9a98200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:28:49.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:48.860+0000 7f0fc9a98200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:28:49.504 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:49.039+0000 7f0fc9a98200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:28:50.276 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:49.990+0000 7f0fc9a98200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:28:50.276 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:50.120+0000 7f0fc9a98200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:28:50.750 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:50.462+0000 7f0fc9a98200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:28:50.751 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:50.553+0000 7f0fc9a98200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:28:50.751 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:50.750+0000 7f0fc9a98200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:51.026 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:50.938+0000 7f0fc9a98200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:28:51.484 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:51.226+0000 7f0fc9a98200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:28:51.484 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:51.483+0000 7f0fc9a98200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:28:51.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:51.721+0000 7f0fc9a98200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:28:52.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:52 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:52.506+0000 7f0fc9a98200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:28:53.380 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:53.070+0000 7f0fc9a98200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:28:53.380 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:53.167+0000 7f0fc9a98200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:28:53.380 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:28:53.380+0000 7f0fc9a98200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:28:53.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: Active manager daemon x restarted 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: Activating manager daemon x 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: mgrmap e10: x(active, starting, since 0.00355582s) 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: Manager daemon x is now available 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:28:54.725 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:28:54.725 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:28:54.725 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:28:54.725 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:54 smithi145 ceph-mon[32151]: mgrmap e11: x(active, since 1.00647s) 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.897+0000 7f87a6f2d700 1 Processor -- start 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.897+0000 7f87a6f2d700 1 -- start start 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.897+0000 7f87a6f2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.897+0000 7f87a6f2d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a007be30 con 0x7f87a0108570 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f87a4cc9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:54.968 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f87a4cc9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60384/0 (socket says 172.21.15.145:60384) 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f87a4cc9700 1 -- 172.21.15.145:0/2927654936 learned_addr learned my addr 172.21.15.145:0/2927654936 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f87a4cc9700 1 -- 172.21.15.145:0/2927654936 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a010aea0 con 0x7f87a0108570 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f87a4cc9700 1 --2- 172.21.15.145:0/2927654936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87940098a0 tx=0x7f8794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506e91fad4aafca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f879f7fe700 1 -- 172.21.15.145:0/2927654936 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8794004160 con 0x7f87a0108570 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f879f7fe700 1 -- 172.21.15.145:0/2927654936 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8794004300 con 0x7f87a0108570 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.898+0000 7f879f7fe700 1 -- 172.21.15.145:0/2927654936 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8794011b20 con 0x7f87a0108570 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 -- 172.21.15.145:0/2927654936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 msgr2=0x7f87a010a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/2927654936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f87940098a0 tx=0x7f8794009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 -- 172.21.15.145:0/2927654936 shutdown_connections 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/2927654936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a010a960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 -- 172.21.15.145:0/2927654936 >> 172.21.15.145:0/2927654936 conn(0x7f87a0101f20 msgr2=0x7f87a0104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 -- 172.21.15.145:0/2927654936 shutdown_connections 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.899+0000 7f87a6f2d700 1 -- 172.21.15.145:0/2927654936 wait complete. 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a6f2d700 1 Processor -- start 2024-03-28T18:28:54.969 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a6f2d700 1 -- start start 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a6f2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a6f2d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a010bea0 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a4cc9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a4cc9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60390/0 (socket says 172.21.15.145:60390) 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a4cc9700 1 -- 172.21.15.145:0/3465361880 learned_addr learned my addr 172.21.15.145:0/3465361880 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a4cc9700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a019bf50 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a4cc9700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8794011b60 tx=0x7f8794011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8794003b60 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f879401a910 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.900+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a019c1e0 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.901+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8794024930 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.901+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a019c640 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.901+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f8794022020 con 0x7f87a0108570 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.902+0000 7f879dffb700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.902+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87a0069250 con 0x7f8790038fd0 2024-03-28T18:28:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.902+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f879404e030 con 0x7f87a0108570 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.902+0000 7f879ffff700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:48.902+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:49.103+0000 7f879ffff700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:49.103+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:49.504+0000 7f879ffff700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:49.504+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:50.305+0000 7f879ffff700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:50.305+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:51.906+0000 7f879ffff700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.145:6800/2089479814 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:51.906+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:53.395+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f879402d770 con 0x7f87a0108570 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:53.395+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 msgr2=0x7f879003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:53.395+0000 7f879dffb700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.397+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f879404f1e0 con 0x7f87a0108570 2024-03-28T18:28:54.971 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.397+0000 7f879dffb700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 0x7f879003ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.397+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8794021e70 con 0x7f879003c890 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.399+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 0x7f879003ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.399+0000 7f879ffff700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 0x7f879003ec80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f878c003a10 tx=0x7f878c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.400+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f8794021e70 con 0x7f879003c890 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.406+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f87a01030d0 con 0x7f879003c890 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f879dffb700 1 -- 172.21.15.145:0/3465361880 <== mgr.14152 v2:172.21.15.145:6800/82375532 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f87a01030d0 con 0x7f879003c890 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 msgr2=0x7f879003ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 0x7f879003ec80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f878c003a10 tx=0x7f878c009270 comp rx=0 tx=0).stop 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 msgr2=0x7f87a019ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8794011b60 tx=0x7f8794011c40 comp rx=0 tx=0).stop 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 shutdown_connections 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f879003c890 0x7f879003ec80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87a0108570 0x7f87a019ba10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 --2- 172.21.15.145:0/3465361880 >> [v2:172.21.15.145:6800/2089479814,v1:172.21.15.145:6801/2089479814] conn(0x7f8790038fd0 0x7f879003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 >> 172.21.15.145:0/3465361880 conn(0x7f87a0101f20 msgr2=0x7f87a01029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 shutdown_connections 2024-03-28T18:28:54.972 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:54.407+0000 7f87a6f2d700 1 -- 172.21.15.145:0/3465361880 wait complete. 2024-03-28T18:28:54.973 INFO:teuthology.orchestra.run.smithi145.stdout:mgr epoch 9 is available 2024-03-28T18:28:54.973 INFO:teuthology.orchestra.run.smithi145.stdout:Generating a dashboard self-signed certificate... 2024-03-28T18:28:55.689 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:55 smithi145 ceph-mon[32151]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T18:28:55.689 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:55 smithi145 ceph-mon[32151]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d2f44700 1 Processor -- start 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d2f44700 1 -- start start 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d2f44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d2f44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc07c820 con 0x7f27cc07de30 2024-03-28T18:28:56.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d0ce0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d0ce0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59128/0 (socket says 172.21.15.145:59128) 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d0ce0700 1 -- 172.21.15.145:0/4261551294 learned_addr learned my addr 172.21.15.145:0/4261551294 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.369+0000 7f27d0ce0700 1 -- 172.21.15.145:0/4261551294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc07c9a0 con 0x7f27cc07de30 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.370+0000 7f27d0ce0700 1 --2- 172.21.15.145:0/4261551294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27c00098a0 tx=0x7f27c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c283807384e7464 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.370+0000 7f27cb7fe700 1 -- 172.21.15.145:0/4261551294 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27c0004160 con 0x7f27cc07de30 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.370+0000 7f27cb7fe700 1 -- 172.21.15.145:0/4261551294 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27c0004300 con 0x7f27cc07de30 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.370+0000 7f27cb7fe700 1 -- 172.21.15.145:0/4261551294 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27c0011b20 con 0x7f27cc07de30 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- 172.21.15.145:0/4261551294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 msgr2=0x7f27cc07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 --2- 172.21.15.145:0/4261551294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27c00098a0 tx=0x7f27c0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- 172.21.15.145:0/4261551294 shutdown_connections 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 --2- 172.21.15.145:0/4261551294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc07c2e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- 172.21.15.145:0/4261551294 >> 172.21.15.145:0/4261551294 conn(0x7f27cc103130 msgr2=0x7f27cc105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- 172.21.15.145:0/4261551294 shutdown_connections 2024-03-28T18:28:56.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- 172.21.15.145:0/4261551294 wait complete. 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 Processor -- start 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.371+0000 7f27d2f44700 1 -- start start 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d2f44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d2f44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc10e130 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d0ce0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d0ce0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59144/0 (socket says 172.21.15.145:59144) 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d0ce0700 1 -- 172.21.15.145:0/3239883232 learned_addr learned my addr 172.21.15.145:0/3239883232 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d0ce0700 1 -- 172.21.15.145:0/3239883232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc19e190 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d0ce0700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27c00116a0 tx=0x7f27c0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27c0003a40 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27c0003be0 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc19e420 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27c001a750 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.372+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc19e880 con 0x7f27cc07de30 2024-03-28T18:28:56.377 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.373+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f27c001aa50 con 0x7f27cc07de30 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.373+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc069250 con 0x7f27cc07de30 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.373+0000 7f27c9ffb700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 0x7f27bc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.374+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27c004d6c0 con 0x7f27cc07de30 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.374+0000 7f27cbfff700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 0x7f27bc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.374+0000 7f27cbfff700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 0x7f27bc03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27b8006fd0 tx=0x7f27b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.379+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f27c0011e20 con 0x7f27cc07de30 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.449+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f27c002d910 con 0x7f27cc07de30 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.605+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f27cc104330 con 0x7f27bc038e10 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.688+0000 7f27c9ffb700 1 -- 172.21.15.145:0/3239883232 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f27cc104330 con 0x7f27bc038e10 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 msgr2=0x7f27bc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 0x7f27bc03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27b8006fd0 tx=0x7f27b8006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 msgr2=0x7f27cc19dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27c00116a0 tx=0x7f27c0011ba0 comp rx=0 tx=0).stop 2024-03-28T18:28:56.378 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 shutdown_connections 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f27bc038e10 0x7f27bc03b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 --2- 172.21.15.145:0/3239883232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc07de30 0x7f27cc19dc50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 >> 172.21.15.145:0/3239883232 conn(0x7f27cc103130 msgr2=0x7f27cc103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 shutdown_connections 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:55.691+0000 7f27d2f44700 1 -- 172.21.15.145:0/3239883232 wait complete. 2024-03-28T18:28:56.379 INFO:teuthology.orchestra.run.smithi145.stdout:Creating initial admin user... 2024-03-28T18:28:56.703 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:54] ENGINE Bus STARTING 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:54] ENGINE Serving on https://172.21.15.145:7150 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:55] ENGINE Serving on http://172.21.15.145:8765 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: [28/Mar/2024:18:28:55] ENGINE Bus STARTED 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: mgrmap e12: x(active, since 2s) 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:28:56.704 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:28:57.733 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:57 smithi145 ceph-mon[32151]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:57.734 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:28:57.734 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Jux7Zcn0F7kYGeUn8/HU/OGciFMLgBYvS6s64szMmYvWVtfYn08CC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711650537, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7d4c07700 1 Processor -- start 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7d4c07700 1 -- start start 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7d4c07700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7d4c07700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0106610 con 0x7fb7d01080d0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7ce59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7ce59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59150/0 (socket says 172.21.15.145:59150) 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.731+0000 7fb7ce59c700 1 -- 172.21.15.145:0/306372788 learned_addr learned my addr 172.21.15.145:0/306372788 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7ce59c700 1 -- 172.21.15.145:0/306372788 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d0106790 con 0x7fb7d01080d0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7ce59c700 1 --2- 172.21.15.145:0/306372788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00098a0 tx=0x7fb7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7637846895034e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7cd59a700 1 -- 172.21.15.145:0/306372788 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c0004160 con 0x7fb7d01080d0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7cd59a700 1 -- 172.21.15.145:0/306372788 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c0004300 con 0x7fb7d01080d0 2024-03-28T18:28:57.741 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7cd59a700 1 -- 172.21.15.145:0/306372788 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c0011b20 con 0x7fb7d01080d0 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7d4c07700 1 -- 172.21.15.145:0/306372788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 msgr2=0x7fb7d01084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.732+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/306372788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00098a0 tx=0x7fb7c0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 -- 172.21.15.145:0/306372788 shutdown_connections 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/306372788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d01084f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 -- 172.21.15.145:0/306372788 >> 172.21.15.145:0/306372788 conn(0x7fb7d0101f90 msgr2=0x7fb7d01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 -- 172.21.15.145:0/306372788 shutdown_connections 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 -- 172.21.15.145:0/306372788 wait complete. 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 Processor -- start 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.733+0000 7fb7d4c07700 1 -- start start 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7d4c07700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7d4c07700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d010afa0 con 0x7fb7d01080d0 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7ce59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7ce59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59160/0 (socket says 172.21.15.145:59160) 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7ce59c700 1 -- 172.21.15.145:0/3553867860 learned_addr learned my addr 172.21.15.145:0/3553867860 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:57.742 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7ce59c700 1 -- 172.21.15.145:0/3553867860 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d019c0a0 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7ce59c700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0000c00 tx=0x7fb7c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c0003b60 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c001a910 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d019c330 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7c0024930 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.734+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d019c790 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.735+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb7c0022020 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.735+0000 7fb7bf7fe700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 0x7fb7b803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.735+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb7c004d330 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.735+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d0069250 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.736+0000 7fb7cdd9b700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 0x7fb7b803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.736+0000 7fb7cdd9b700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 0x7fb7b803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4006fd0 tx=0x7fb7c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.741+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7c000be30 con 0x7fb7d01080d0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:56.955+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d002dca0 con 0x7fb7b8038fd0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.181+0000 7fb7bf7fe700 1 -- 172.21.15.145:0/3553867860 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb7d002dca0 con 0x7fb7b8038fd0 2024-03-28T18:28:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 msgr2=0x7fb7b803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 0x7fb7b803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4006fd0 tx=0x7fb7c4006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 msgr2=0x7fb7d019bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0000c00 tx=0x7fb7c0011c10 comp rx=0 tx=0).stop 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 shutdown_connections 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7b8038fd0 0x7fb7b803b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.183+0000 7fb7d4c07700 1 --2- 172.21.15.145:0/3553867860 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7d01080d0 0x7fb7d019bb60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.184+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 >> 172.21.15.145:0/3553867860 conn(0x7fb7d0101f90 msgr2=0x7fb7d0102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.184+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 shutdown_connections 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:57.184+0000 7fb7d4c07700 1 -- 172.21.15.145:0/3553867860 wait complete. 2024-03-28T18:28:57.744 INFO:teuthology.orchestra.run.smithi145.stdout:Fetching dashboard port number... 2024-03-28T18:28:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:58 smithi145 ceph-mon[32151]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:28:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:28:58 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/49511741' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.109+0000 7ff498920700 1 Processor -- start 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.109+0000 7ff498920700 1 -- start start 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff498920700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff498920700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49407c8f0 con 0x7ff49407dec0 2024-03-28T18:28:59.006 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59172/0 (socket says 172.21.15.145:59172) 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49259c700 1 -- 172.21.15.145:0/1254465540 learned_addr learned my addr 172.21.15.145:0/1254465540 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49259c700 1 -- 172.21.15.145:0/1254465540 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49407ca70 con 0x7ff49407dec0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49259c700 1 --2- 172.21.15.145:0/1254465540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4880098a0 tx=0x7ff488009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef85162056e55fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49159a700 1 -- 172.21.15.145:0/1254465540 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff488004160 con 0x7ff49407dec0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49159a700 1 -- 172.21.15.145:0/1254465540 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff488004300 con 0x7ff49407dec0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.110+0000 7ff49159a700 1 -- 172.21.15.145:0/1254465540 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff488011b20 con 0x7ff49407dec0 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 -- 172.21.15.145:0/1254465540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 msgr2=0x7ff49407c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 --2- 172.21.15.145:0/1254465540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4880098a0 tx=0x7ff488009bb0 comp rx=0 tx=0).stop 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 -- 172.21.15.145:0/1254465540 shutdown_connections 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 --2- 172.21.15.145:0/1254465540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49407c320 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 -- 172.21.15.145:0/1254465540 >> 172.21.15.145:0/1254465540 conn(0x7ff494103050 msgr2=0x7ff494105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 -- 172.21.15.145:0/1254465540 shutdown_connections 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.111+0000 7ff498920700 1 -- 172.21.15.145:0/1254465540 wait complete. 2024-03-28T18:28:59.007 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff498920700 1 Processor -- start 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff498920700 1 -- start start 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff498920700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff498920700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49410e0d0 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff49259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff49259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59176/0 (socket says 172.21.15.145:59176) 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff49259c700 1 -- 172.21.15.145:0/49511741 learned_addr learned my addr 172.21.15.145:0/49511741 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff49259c700 1 -- 172.21.15.145:0/49511741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49419e120 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff49259c700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4880117e0 tx=0x7ff488011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff488003bf0 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.112+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff48801a910 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.113+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49419e3b0 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.113+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff488024970 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.113+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49419e810 con 0x7ff49407dec0 2024-03-28T18:28:59.008 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ff488022020 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4940637c0 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff4837fe700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 0x7ff47c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff48804d8a0 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff491d9b700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 0x7ff47c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.114+0000 7ff491d9b700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 0x7ff47c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff484006fd0 tx=0x7ff484006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.119+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff48802cdf0 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.322+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff4941a11e0 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.323+0000 7ff4837fe700 1 -- 172.21.15.145:0/49511741 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff48801ab30 con 0x7ff49407dec0 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 msgr2=0x7ff47c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 0x7ff47c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff484006fd0 tx=0x7ff484006e40 comp rx=0 tx=0).stop 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 msgr2=0x7ff49419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4880117e0 tx=0x7ff488011ba0 comp rx=0 tx=0).stop 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 shutdown_connections 2024-03-28T18:28:59.009 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff47c038f70 0x7ff47c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:59.010 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 --2- 172.21.15.145:0/49511741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff49407dec0 0x7ff49419dbe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:28:59.010 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 >> 172.21.15.145:0/49511741 conn(0x7ff494103050 msgr2=0x7ff494103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:28:59.010 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 shutdown_connections 2024-03-28T18:28:59.010 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:58.327+0000 7ff498920700 1 -- 172.21.15.145:0/49511741 wait complete. 2024-03-28T18:28:59.013 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:28:59.013 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout disabled 2024-03-28T18:28:59.021 INFO:teuthology.orchestra.run.smithi145.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:28:59.021 INFO:teuthology.orchestra.run.smithi145.stdout:systemctl: stdout inactive 2024-03-28T18:28:59.021 INFO:teuthology.orchestra.run.smithi145.stdout:firewalld.service is not enabled 2024-03-28T18:28:59.021 INFO:teuthology.orchestra.run.smithi145.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout:Ceph Dashboard is now available at: 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout: URL: https://smithi145.front.sepia.ceph.com:8443/ 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout: User: admin 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout: Password: ypysehqtoy 2024-03-28T18:28:59.023 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:28:59.024 INFO:teuthology.orchestra.run.smithi145.stdout:Saving cluster configuration to /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config directory 2024-03-28T18:28:59.024 INFO:teuthology.orchestra.run.smithi145.stdout:Enabling autotune for osd_memory_target 2024-03-28T18:29:00.365 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:00.372 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.448+0000 7fbf88e44700 1 Processor -- start 2024-03-28T18:29:00.372 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf88e44700 1 -- start start 2024-03-28T18:29:00.372 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf88e44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:00.372 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf88e44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8407be30 con 0x7fbf84106380 2024-03-28T18:29:00.372 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf8259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf8259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59178/0 (socket says 172.21.15.145:59178) 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf8259c700 1 -- 172.21.15.145:0/3218431667 learned_addr learned my addr 172.21.15.145:0/3218431667 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.449+0000 7fbf8259c700 1 -- 172.21.15.145:0/3218431667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf84106ce0 con 0x7fbf84106380 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf8259c700 1 --2- 172.21.15.145:0/3218431667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf740098a0 tx=0x7fbf74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf5885eb9c2a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf8159a700 1 -- 172.21.15.145:0/3218431667 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf74004160 con 0x7fbf84106380 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf8159a700 1 -- 172.21.15.145:0/3218431667 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf74004300 con 0x7fbf84106380 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf8159a700 1 -- 172.21.15.145:0/3218431667 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf74011b20 con 0x7fbf84106380 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf88e44700 1 -- 172.21.15.145:0/3218431667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 msgr2=0x7fbf841067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf88e44700 1 --2- 172.21.15.145:0/3218431667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf740098a0 tx=0x7fbf74009bb0 comp rx=0 tx=0).stop 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf88e44700 1 -- 172.21.15.145:0/3218431667 shutdown_connections 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf88e44700 1 --2- 172.21.15.145:0/3218431667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf841067a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.450+0000 7fbf88e44700 1 -- 172.21.15.145:0/3218431667 >> 172.21.15.145:0/3218431667 conn(0x7fbf84101f60 msgr2=0x7fbf84104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 -- 172.21.15.145:0/3218431667 shutdown_connections 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 -- 172.21.15.145:0/3218431667 wait complete. 2024-03-28T18:29:00.373 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 Processor -- start 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 -- start start 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.451+0000 7fbf88e44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf84107db0 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf8259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf8259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59180/0 (socket says 172.21.15.145:59180) 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf8259c700 1 -- 172.21.15.145:0/2994825606 learned_addr learned my addr 172.21.15.145:0/2994825606 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf8259c700 1 -- 172.21.15.145:0/2994825606 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8419e110 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf8259c700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf740117e0 tx=0x7fbf74011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf74003ba0 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8419e3a0 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8419e800 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf7401a910 con 0x7fbf84106380 2024-03-28T18:29:00.374 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.452+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf740248f0 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.453+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fbf74022020 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.453+0000 7fbf737fe700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 0x7fbf6c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.453+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbf7404e720 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.454+0000 7fbf81d9b700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 0x7fbf6c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.453+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf64005360 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.454+0000 7fbf81d9b700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 0x7fbf6c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf78006fd0 tx=0x7fbf78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.463+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf740172b0 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.669+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fbf64005650 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.670+0000 7fbf737fe700 1 -- 172.21.15.145:0/2994825606 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fbf7401aac0 con 0x7fbf84106380 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 msgr2=0x7fbf6c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 0x7fbf6c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbf78006fd0 tx=0x7fbf78006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 msgr2=0x7fbf8419dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf740117e0 tx=0x7fbf74011ba0 comp rx=0 tx=0).stop 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 shutdown_connections 2024-03-28T18:29:00.375 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf6c038fd0 0x7fbf6c03b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:00.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 --2- 172.21.15.145:0/2994825606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf84106380 0x7fbf8419dbd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:00.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 >> 172.21.15.145:0/2994825606 conn(0x7fbf84101f60 msgr2=0x7fbf84194840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:00.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 shutdown_connections 2024-03-28T18:29:00.376 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:28:59.672+0000 7fbf88e44700 1 -- 172.21.15.145:0/2994825606 wait complete. 2024-03-28T18:29:01.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:01 smithi145 ceph-mon[32151]: mgrmap e13: x(active, since 6s) 2024-03-28T18:29:01.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:01 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/272320674' entity='client.admin' 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.734+0000 7f5847f78700 1 Processor -- start 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5847f78700 1 -- start start 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5847f78700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5847f78700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401078b0 con 0x7f5840106ec0 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5845d14700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5845d14700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59194/0 (socket says 172.21.15.145:59194) 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5845d14700 1 -- 172.21.15.145:0/3192098968 learned_addr learned my addr 172.21.15.145:0/3192098968 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:01.690 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5845d14700 1 -- 172.21.15.145:0/3192098968 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840107be0 con 0x7f5840106ec0 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5845d14700 1 --2- 172.21.15.145:0/3192098968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098a0 tx=0x7f583c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6880ac538d623e13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5844d12700 1 -- 172.21.15.145:0/3192098968 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f583c004160 con 0x7f5840106ec0 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5844d12700 1 -- 172.21.15.145:0/3192098968 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583c004300 con 0x7f5840106ec0 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.735+0000 7f5844d12700 1 -- 172.21.15.145:0/3192098968 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f583c011b20 con 0x7f5840106ec0 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 -- 172.21.15.145:0/3192098968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 msgr2=0x7f58401072e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 --2- 172.21.15.145:0/3192098968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098a0 tx=0x7f583c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 -- 172.21.15.145:0/3192098968 shutdown_connections 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 --2- 172.21.15.145:0/3192098968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f58401072e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 -- 172.21.15.145:0/3192098968 >> 172.21.15.145:0/3192098968 conn(0x7f5840101740 msgr2=0x7f5840103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 -- 172.21.15.145:0/3192098968 shutdown_connections 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.736+0000 7f5847f78700 1 -- 172.21.15.145:0/3192098968 wait complete. 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5847f78700 1 Processor -- start 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5847f78700 1 -- start start 2024-03-28T18:29:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5847f78700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5847f78700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584010af90 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5845d14700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5845d14700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59198/0 (socket says 172.21.15.145:59198) 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5845d14700 1 -- 172.21.15.145:0/272320674 learned_addr learned my addr 172.21.15.145:0/272320674 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5845d14700 1 -- 172.21.15.145:0/272320674 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58401920d0 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5845d14700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f583c0116a0 tx=0x7f583c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.737+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f583c003a40 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.738+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583c003be0 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.738+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f583c01a750 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.738+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58401902c0 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.738+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5840190780 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.738+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f583c01aa00 con 0x7f5840106ec0 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.739+0000 7f5836ffd700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 0x7f582c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:01.692 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.739+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f583c04d610 con 0x7f5840106ec0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.739+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58400637c0 con 0x7f5840106ec0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.739+0000 7f5845513700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 0x7f582c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.739+0000 7f5845513700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 0x7f582c03b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5830006fd0 tx=0x7f5830006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:00.744+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f583c011e20 con 0x7f5840106ec0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.054+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f5840190a00 con 0x7f5840106ec0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.056+0000 7f5836ffd700 1 -- 172.21.15.145:0/272320674 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) v1 ==== 153+0+0 (secure 0 0 0) 0x7f583c032030 con 0x7f5840106ec0 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 msgr2=0x7f582c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 0x7f582c03b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5830006fd0 tx=0x7f5830006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 msgr2=0x7f5840191b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f583c0116a0 tx=0x7f583c011ba0 comp rx=0 tx=0).stop 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 shutdown_connections 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f582c039020 0x7f582c03b4e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 --2- 172.21.15.145:0/272320674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5840106ec0 0x7f5840191b90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 >> 172.21.15.145:0/272320674 conn(0x7f5840101740 msgr2=0x7f58401022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:01.693 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 shutdown_connections 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr 2024-03-28T18:29:01.059+0000 7f5847f78700 1 -- 172.21.15.145:0/272320674 wait complete. 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: ceph telemetry on 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout:For more information see: 2024-03-28T18:29:01.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.695 INFO:teuthology.orchestra.run.smithi145.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T18:29:01.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:01.695 INFO:teuthology.orchestra.run.smithi145.stdout:Bootstrap complete. 2024-03-28T18:29:01.730 INFO:tasks.cephadm:Fetching config... 2024-03-28T18:29:01.731 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:01.731 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T18:29:01.746 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T18:29:01.746 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:01.747 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T18:29:01.801 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T18:29:01.801 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:01.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T18:29:01.869 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T18:29:01.869 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:01.869 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T18:29:01.925 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T18:29:01.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:29:02.003 INFO:teuthology.orchestra.run.smithi145.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:02.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:29:02.062 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:02.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:29:02.122 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgdDQNfi0nv6l1aa/Z1e9vtm1Gk6xMaSkx1kElQgGU68OueOSlYDDancYbVaceWYVZALZP6KwnroVaWjOCqvAIhC8r/SPM+KwOILbU3GhCHercb3dQ//jSIuVYc61ZPe5tmP8NMvYmMwfr4W9VzIMaaUp7Bp9HdflzB+X9L0b9e/wTMtH/323yYuOPL4fncASgtO5vpLjv3xL+3OibzDDctJukJn2ATgCzS0XcBT2TRRgNDmxgCIP86Bhmbj1ommLAMS+9jVZQ2EdhxtCZnO415dzviPEgvgIILOZhFPc2TuVe4sgX4xjn+wBbewjRl+Kt1u1pTz+jTwnBEF7Rx55Aec3daFWaGbqU5eOQyCrkYqNhQaWvr52r0zJPJTymBaxAiEOedm9zvrtAuzVOzjEXyGPmPAzufSSbaMeO/xZ1XJmHzKJlZqy0gAirf5s82ImPW4FIbWlBkzXVCcm4l4w/o8CfY16BEHep9x8TUyOQ32e0nEmeApGUpCMQbpG2LB0= ceph-b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:02.137 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T18:29:02.588 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:04.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 -- 172.21.15.145:0/2548794501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 msgr2=0x7f5bd8104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:04.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/2548794501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8104a50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009910 tx=0x7f5bc8009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:04.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 -- 172.21.15.145:0/2548794501 shutdown_connections 2024-03-28T18:29:04.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/2548794501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8104a50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 -- 172.21.15.145:0/2548794501 >> 172.21.15.145:0/2548794501 conn(0x7f5bd80fc090 msgr2=0x7f5bd80fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 -- 172.21.15.145:0/2548794501 shutdown_connections 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 -- 172.21.15.145:0/2548794501 wait complete. 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.773+0000 7f5bddb8b700 1 Processor -- start 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bddb8b700 1 -- start start 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bddb8b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bddb8b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd8107c00 con 0x7f5bd8102660 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd77fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd77fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58084/0 (socket says 172.21.15.145:58084) 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd77fe700 1 -- 172.21.15.145:0/1820629516 learned_addr learned my addr 172.21.15.145:0/1820629516 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd77fe700 1 -- 172.21.15.145:0/1820629516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8102420 con 0x7f5bd8102660 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd77fe700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005b40 tx=0x7f5bc8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc8027070 con 0x7f5bd8102660 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bc800be40 con 0x7f5bd8102660 2024-03-28T18:29:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd8100610 con 0x7f5bd8102660 2024-03-28T18:29:04.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc800f460 con 0x7f5bd8102660 2024-03-28T18:29:04.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.774+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd8100b50 con 0x7f5bd8102660 2024-03-28T18:29:04.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.775+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5bc802a070 con 0x7f5bd8102660 2024-03-28T18:29:04.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.775+0000 7f5bd4ff9700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 0x7f5bc003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.775+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5bc801e030 con 0x7f5bd8102660 2024-03-28T18:29:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.775+0000 7f5bd6ffd700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 0x7f5bc003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.775+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd8108380 con 0x7f5bd8102660 2024-03-28T18:29:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.776+0000 7f5bd6ffd700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 0x7f5bc003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006fd0 tx=0x7f5bcc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:04.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.780+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5bc8013070 con 0x7f5bd8102660 2024-03-28T18:29:04.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.976+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5bd81905c0 con 0x7f5bd8102660 2024-03-28T18:29:04.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.978+0000 7f5bd4ff9700 1 -- 172.21.15.145:0/1820629516 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5bc8031020 con 0x7f5bd8102660 2024-03-28T18:29:04.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.980+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 msgr2=0x7f5bc003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.980+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 0x7f5bc003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006fd0 tx=0x7f5bcc006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.980+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 msgr2=0x7f5bd8101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.980+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005b40 tx=0x7f5bc8005030 comp rx=0 tx=0).stop 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.980+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 shutdown_connections 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.981+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5bc0038fc0 0x7f5bc003b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.981+0000 7f5bddb8b700 1 --2- 172.21.15.145:0/1820629516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bd8102660 0x7f5bd8101ee0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.981+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 >> 172.21.15.145:0/1820629516 conn(0x7f5bd80fc090 msgr2=0x7f5bd80fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.981+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 shutdown_connections 2024-03-28T18:29:04.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:04.981+0000 7f5bddb8b700 1 -- 172.21.15.145:0/1820629516 wait complete. 2024-03-28T18:29:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:05.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:05.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:05.576 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T18:29:05.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T18:29:05.811 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:06.201 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1820629516' entity='client.admin' 2024-03-28T18:29:06.201 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:06.201 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:07.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.204+0000 7f47bfa2c700 1 -- 172.21.15.145:0/588240487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 msgr2=0x7f47b8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:07.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.204+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/588240487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b8100ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009910 tx=0x7f47b4009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:07.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.204+0000 7f47bfa2c700 1 -- 172.21.15.145:0/588240487 shutdown_connections 2024-03-28T18:29:07.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.204+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/588240487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b8100ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:07.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.204+0000 7f47bfa2c700 1 -- 172.21.15.145:0/588240487 >> 172.21.15.145:0/588240487 conn(0x7f47b80fc080 msgr2=0x7f47b80fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 -- 172.21.15.145:0/588240487 shutdown_connections 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 -- 172.21.15.145:0/588240487 wait complete. 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 Processor -- start 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 -- start start 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bfa2c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b8106b50 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bd7c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bd7c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58104/0 (socket says 172.21.15.145:58104) 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.205+0000 7f47bd7c8700 1 -- 172.21.15.145:0/374019332 learned_addr learned my addr 172.21.15.145:0/374019332 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47bd7c8700 1 -- 172.21.15.145:0/374019332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b819a050 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47bd7c8700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47b4006010 tx=0x7f47b4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b4027070 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b400be40 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b819a220 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b400f460 con 0x7f47b8100ac0 2024-03-28T18:29:07.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.206+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b819a5c0 con 0x7f47b8100ac0 2024-03-28T18:29:07.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f47b402a070 con 0x7f47b8100ac0 2024-03-28T18:29:07.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b81072d0 con 0x7f47b8100ac0 2024-03-28T18:29:07.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47ae7fc700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 0x7f47a403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:07.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f47b401e030 con 0x7f47b8100ac0 2024-03-28T18:29:07.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47bcfc7700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 0x7f47a403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:07.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.207+0000 7f47bcfc7700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 0x7f47a403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f47a8006fd0 tx=0x7f47a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:07.212 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.212+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f47b4013070 con 0x7f47b8100ac0 2024-03-28T18:29:07.414 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.414+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b802dc30 con 0x7f47a4039020 2024-03-28T18:29:07.417 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.417+0000 7f47ae7fc700 1 -- 172.21.15.145:0/374019332 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f47b802dc30 con 0x7f47a4039020 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 msgr2=0x7f47a403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 0x7f47a403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f47a8006fd0 tx=0x7f47a8006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 msgr2=0x7f47b806aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47b4006010 tx=0x7f47b4005050 comp rx=0 tx=0).stop 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 shutdown_connections 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f47a4039020 0x7f47a403b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 --2- 172.21.15.145:0/374019332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47b8100ac0 0x7f47b806aa00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.419+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 >> 172.21.15.145:0/374019332 conn(0x7f47b80fc080 msgr2=0x7f47b80fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.420+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 shutdown_connections 2024-03-28T18:29:07.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:07.420+0000 7f47bfa2c700 1 -- 172.21.15.145:0/374019332 wait complete. 2024-03-28T18:29:08.160 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi164 2024-03-28T18:29:08.160 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:29:08.160 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:29:08.179 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:29:08.179 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:08.236 INFO:tasks.cephadm:Adding host smithi164 to orchestrator... 2024-03-28T18:29:08.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch host add smithi164 2024-03-28T18:29:08.473 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:08.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:09.711 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:09 smithi145 ceph-mon[32151]: Updating smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:09.711 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:09 smithi145 ceph-mon[32151]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:09.711 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:09 smithi145 ceph-mon[32151]: Updating smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:09.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 -- 172.21.15.145:0/3593370515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 msgr2=0x7f9e9c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:09.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 --2- 172.21.15.145:0/3593370515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1021a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b0a0 tx=0x7f9e8c00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:29:09.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 -- 172.21.15.145:0/3593370515 shutdown_connections 2024-03-28T18:29:09.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 --2- 172.21.15.145:0/3593370515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1021a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:09.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 -- 172.21.15.145:0/3593370515 >> 172.21.15.145:0/3593370515 conn(0x7f9e9c0fd670 msgr2=0x7f9e9c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 -- 172.21.15.145:0/3593370515 shutdown_connections 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.898+0000 7f9ea11be700 1 -- 172.21.15.145:0/3593370515 wait complete. 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9ea11be700 1 Processor -- start 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9ea11be700 1 -- start start 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9ea11be700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9ea11be700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c10c190 con 0x7f9e9c101dc0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9e9ad9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9e9ad9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58112/0 (socket says 172.21.15.145:58112) 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.899+0000 7f9e9ad9d700 1 -- 172.21.15.145:0/1452970403 learned_addr learned my addr 172.21.15.145:0/1452970403 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9e9ad9d700 1 -- 172.21.15.145:0/1452970403 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c197b30 con 0x7f9e9c101dc0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9e9ad9d700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c015040 tx=0x7f9e8c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e8c017040 con 0x7f9e9c101dc0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c022070 con 0x7f9e9c101dc0 2024-03-28T18:29:09.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c197dc0 con 0x7f9e9c101dc0 2024-03-28T18:29:09.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e8c01d5d0 con 0x7f9e9c101dc0 2024-03-28T18:29:09.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.900+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c19bec0 con 0x7f9e9c101dc0 2024-03-28T18:29:09.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f9e8c01d880 con 0x7f9e9c101dc0 2024-03-28T18:29:09.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9e93fff700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 0x7f9e8803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:09.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c1032a0 con 0x7f9e9c101dc0 2024-03-28T18:29:09.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9e8c00f050 con 0x7f9e9c101dc0 2024-03-28T18:29:09.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9e9a59c700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 0x7f9e8803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:09.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.901+0000 7f9e9a59c700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 0x7f9e8803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e84006fd0 tx=0x7f9e84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:09.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:09.906+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e8c007e20 con 0x7f9e9c101dc0 2024-03-28T18:29:10.106 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:10.105+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e9c0637c0 con 0x7f9e88039020 2024-03-28T18:29:11.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:10 smithi145 ceph-mon[32151]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:12.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:11 smithi145 ceph-mon[32151]: Deploying cephadm binary to smithi164 2024-03-28T18:29:14.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.127+0000 7f9e93fff700 1 -- 172.21.15.145:0/1452970403 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9e9c0637c0 con 0x7f9e88039020 2024-03-28T18:29:14.128 INFO:teuthology.orchestra.run.smithi145.stdout:Added host 'smithi164' with addr '172.21.15.164' 2024-03-28T18:29:14.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 msgr2=0x7f9e8803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:14.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 0x7f9e8803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9e84006fd0 tx=0x7f9e84006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 msgr2=0x7f9e9c1975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c015040 tx=0x7f9e8c003d50 comp rx=0 tx=0).stop 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 shutdown_connections 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9e88039020 0x7f9e8803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 --2- 172.21.15.145:0/1452970403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e9c101dc0 0x7f9e9c1975f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.129+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 >> 172.21.15.145:0/1452970403 conn(0x7f9e9c0fd670 msgr2=0x7f9e9c103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.130+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 shutdown_connections 2024-03-28T18:29:14.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:14.130+0000 7f9ea11be700 1 -- 172.21.15.145:0/1452970403 wait complete. 2024-03-28T18:29:14.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:14 smithi145 ceph-mon[32151]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:14.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:14 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:14.685 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:14 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:14.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:29:14.943 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:15.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:15 smithi145 ceph-mon[32151]: Added host smithi164 2024-03-28T18:29:15.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:15.882 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 -- 172.21.15.145:0/1712584796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 msgr2=0x7f8838106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 --2- 172.21.15.145:0/1712584796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f8838106e50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8830009910 tx=0x7f8830009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 -- 172.21.15.145:0/1712584796 shutdown_connections 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 --2- 172.21.15.145:0/1712584796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f8838106e50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 -- 172.21.15.145:0/1712584796 >> 172.21.15.145:0/1712584796 conn(0x7f88380fc560 msgr2=0x7f88380fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 -- 172.21.15.145:0/1712584796 shutdown_connections 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.359+0000 7f883dbda700 1 -- 172.21.15.145:0/1712584796 wait complete. 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f883dbda700 1 Processor -- start 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f883dbda700 1 -- start start 2024-03-28T18:29:16.362 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f883dbda700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f883dbda700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883810b0b0 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f88377fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f88377fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43330/0 (socket says 172.21.15.145:43330) 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f88377fe700 1 -- 172.21.15.145:0/646562843 learned_addr learned my addr 172.21.15.145:0/646562843 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.360+0000 7f88377fe700 1 -- 172.21.15.145:0/646562843 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838198a20 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f88377fe700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8830005850 tx=0x7f8830005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8830026070 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f883000bbd0 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8838198d10 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8830003970 con 0x7f8838106a70 2024-03-28T18:29:16.363 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88381991d0 con 0x7f8838106a70 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.361+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f8830029070 con 0x7f8838106a70 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.362+0000 7f8834ff9700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 0x7f882003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.362+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88381021b0 con 0x7f8838106a70 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.362+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f883001e070 con 0x7f8838106a70 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.362+0000 7f8836ffd700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 0x7f882003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.362+0000 7f8836ffd700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 0x7f882003b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f882c00ad30 tx=0x7f882c0093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:16.367 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.367+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8830013070 con 0x7f8838106a70 2024-03-28T18:29:16.569 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.568+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f883806a570 con 0x7f8820038fc0 2024-03-28T18:29:16.570 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.569+0000 7f8834ff9700 1 -- 172.21.15.145:0/646562843 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f883806a570 con 0x7f8820038fc0 2024-03-28T18:29:16.571 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:16.571 INFO:teuthology.orchestra.run.smithi145.stdout:[{"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}, {"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}] 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 msgr2=0x7f882003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 0x7f882003b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f882c00ad30 tx=0x7f882c0093f0 comp rx=0 tx=0).stop 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 msgr2=0x7f88381984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8830005850 tx=0x7f8830005030 comp rx=0 tx=0).stop 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 shutdown_connections 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8820038fc0 0x7f882003b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 --2- 172.21.15.145:0/646562843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8838106a70 0x7f88381984e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:16.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 >> 172.21.15.145:0/646562843 conn(0x7f88380fc560 msgr2=0x7f88381085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:16.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 shutdown_connections 2024-03-28T18:29:16.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:16.572+0000 7f883dbda700 1 -- 172.21.15.145:0/646562843 wait complete. 2024-03-28T18:29:16.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:16 smithi145 ceph-mon[32151]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:16.880 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:17.166 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-03-28T18:29:17.166 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:29:17.166 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:29:17.184 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:29:17.184 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:17.240 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-03-28T18:29:17.240 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch host add smithi192 2024-03-28T18:29:17.473 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:18.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:17 smithi145 ceph-mon[32151]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 -- 172.21.15.145:0/4290231823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 msgr2=0x7f6a74106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 --2- 172.21.15.145:0/4290231823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a74106e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009910 tx=0x7f6a70009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 -- 172.21.15.145:0/4290231823 shutdown_connections 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 --2- 172.21.15.145:0/4290231823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a74106e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 -- 172.21.15.145:0/4290231823 >> 172.21.15.145:0/4290231823 conn(0x7f6a740fc5a0 msgr2=0x7f6a740fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 -- 172.21.15.145:0/4290231823 shutdown_connections 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 -- 172.21.15.145:0/4290231823 wait complete. 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.830+0000 7f6a7c270700 1 Processor -- start 2024-03-28T18:29:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7c270700 1 -- start start 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7c270700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7c270700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7410b0b0 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7a00c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7a00c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43354/0 (socket says 172.21.15.145:43354) 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7a00c700 1 -- 172.21.15.145:0/3955741969 learned_addr learned my addr 172.21.15.145:0/3955741969 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7a00c700 1 -- 172.21.15.145:0/3955741969 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740fe940 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7a00c700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6a70005850 tx=0x7f6a70005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a70026070 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740fcff0 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a7000bc30 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.831+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a70003a20 con 0x7f6a74106a70 2024-03-28T18:29:18.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.832+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a740fd4b0 con 0x7f6a74106a70 2024-03-28T18:29:18.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.832+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f6a70029070 con 0x7f6a74106a70 2024-03-28T18:29:18.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.832+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a741021b0 con 0x7f6a74106a70 2024-03-28T18:29:18.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.832+0000 7f6a6b7fe700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 0x7f6a6003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:18.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.833+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6a7001e070 con 0x7f6a74106a70 2024-03-28T18:29:18.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.833+0000 7f6a7980b700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 0x7f6a6003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:18.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.833+0000 7f6a7980b700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 0x7f6a6003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6a64006fd0 tx=0x7f6a64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:18.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:18.837+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a70013070 con 0x7f6a74106a70 2024-03-28T18:29:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:18 smithi145 ceph-mon[32151]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:19.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:19.038+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a740fdff0 con 0x7f6a60038fc0 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:20.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:20.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:20.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:21 smithi145 ceph-mon[32151]: Deploying cephadm binary to smithi192 2024-03-28T18:29:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:21 smithi145 ceph-mon[32151]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:21 smithi145 ceph-mon[32151]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:22.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:22 smithi145 ceph-mon[32151]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:23.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.143+0000 7f6a6b7fe700 1 -- 172.21.15.145:0/3955741969 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6a740fdff0 con 0x7f6a60038fc0 2024-03-28T18:29:23.143 INFO:teuthology.orchestra.run.smithi145.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-03-28T18:29:23.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 msgr2=0x7f6a6003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:23.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 0x7f6a6003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6a64006fd0 tx=0x7f6a64006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:23.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 msgr2=0x7f6a740fca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6a70005850 tx=0x7f6a70005030 comp rx=0 tx=0).stop 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 shutdown_connections 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6a60038fc0 0x7f6a6003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 --2- 172.21.15.145:0/3955741969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a74106a70 0x7f6a740fca50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 >> 172.21.15.145:0/3955741969 conn(0x7f6a740fc5a0 msgr2=0x7f6a74069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 shutdown_connections 2024-03-28T18:29:23.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:23.145+0000 7f6a7c270700 1 -- 172.21.15.145:0/3955741969 wait complete. 2024-03-28T18:29:23.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:29:23.948 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:24.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:24.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:24 smithi145 ceph-mon[32151]: Added host smithi192 2024-03-28T18:29:24.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:24.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:24 smithi145 ceph-mon[32151]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:24.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:25.417 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:25.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 -- 172.21.15.145:0/4263694737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 msgr2=0x7f5dbc106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:25.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 --2- 172.21.15.145:0/4263694737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc106f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009910 tx=0x7f5dac009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:25.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 -- 172.21.15.145:0/4263694737 shutdown_connections 2024-03-28T18:29:25.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 --2- 172.21.15.145:0/4263694737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc106f50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:25.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 -- 172.21.15.145:0/4263694737 >> 172.21.15.145:0/4263694737 conn(0x7f5dbc0fbcd0 msgr2=0x7f5dbc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:25.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 -- 172.21.15.145:0/4263694737 shutdown_connections 2024-03-28T18:29:25.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.445+0000 7f5dc2401700 1 -- 172.21.15.145:0/4263694737 wait complete. 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dc2401700 1 Processor -- start 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dc2401700 1 -- start start 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dc2401700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dc2401700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc10c2d0 con 0x7f5dbc106b70 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dbbfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dbbfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41548/0 (socket says 172.21.15.145:41548) 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.446+0000 7f5dbbfff700 1 -- 172.21.15.145:0/2058407160 learned_addr learned my addr 172.21.15.145:0/2058407160 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5dbbfff700 1 -- 172.21.15.145:0/2058407160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc19d940 con 0x7f5dbc106b70 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5dbbfff700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dac005850 tx=0x7f5dac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dac026070 con 0x7f5dbc106b70 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5dac00bc30 con 0x7f5dbc106b70 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dac003a20 con 0x7f5dbc106b70 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc19dbd0 con 0x7f5dbc106b70 2024-03-28T18:29:25.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.447+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc1975c0 con 0x7f5dbc106b70 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.448+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5dac029070 con 0x7f5dbc106b70 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.448+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc1032b0 con 0x7f5dbc106b70 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.448+0000 7f5db97fa700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 0x7f5da403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.448+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5dac01e070 con 0x7f5dbc106b70 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.448+0000 7f5dbb7fe700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 0x7f5da403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:25.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.449+0000 7f5dbb7fe700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 0x7f5da403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006fd0 tx=0x7f5db0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:25.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.453+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5dac013070 con 0x7f5dbc106b70 2024-03-28T18:29:25.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.657+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5dbc0637c0 con 0x7f5da4038fc0 2024-03-28T18:29:25.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.658+0000 7f5db97fa700 1 -- 172.21.15.145:0/2058407160 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f5dbc0637c0 con 0x7f5da4038fc0 2024-03-28T18:29:25.658 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:29:25.658 INFO:teuthology.orchestra.run.smithi145.stdout:[{"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}, {"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 msgr2=0x7f5da403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 0x7f5da403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006fd0 tx=0x7f5db0006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 msgr2=0x7f5dbc19d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dac005850 tx=0x7f5dac005030 comp rx=0 tx=0).stop 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 shutdown_connections 2024-03-28T18:29:25.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5da4038fc0 0x7f5da403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:25.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 --2- 172.21.15.145:0/2058407160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dbc106b70 0x7f5dbc19d400 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:25.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 >> 172.21.15.145:0/2058407160 conn(0x7f5dbc0fbcd0 msgr2=0x7f5dbc0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:25.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 shutdown_connections 2024-03-28T18:29:25.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:25.660+0000 7f5dc2401700 1 -- 172.21.15.145:0/2058407160 wait complete. 2024-03-28T18:29:26.254 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:26 smithi145 ceph-mon[32151]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:26.254 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:26.281 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T18:29:26.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T18:29:26.512 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:27 smithi145 ceph-mon[32151]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.890+0000 7ff2b8daf700 1 -- 172.21.15.145:0/286132962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 msgr2=0x7ff2b4106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.890+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/286132962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4106e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009910 tx=0x7ff2a8009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 -- 172.21.15.145:0/286132962 shutdown_connections 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/286132962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4106e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 -- 172.21.15.145:0/286132962 >> 172.21.15.145:0/286132962 conn(0x7ff2b40fc580 msgr2=0x7ff2b40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 -- 172.21.15.145:0/286132962 shutdown_connections 2024-03-28T18:29:27.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 -- 172.21.15.145:0/286132962 wait complete. 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 Processor -- start 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.891+0000 7ff2b8daf700 1 -- start start 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b8daf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b8daf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b410b090 con 0x7ff2b4106a50 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41568/0 (socket says 172.21.15.145:41568) 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b259c700 1 -- 172.21.15.145:0/2404666326 learned_addr learned my addr 172.21.15.145:0/2404666326 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b259c700 1 -- 172.21.15.145:0/2404666326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b4197bb0 con 0x7ff2b4106a50 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b259c700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8005850 tx=0x7ff2a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8026070 con 0x7ff2b4106a50 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2a800bbd0 con 0x7ff2b4106a50 2024-03-28T18:29:27.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b4197ea0 con 0x7ff2b4106a50 2024-03-28T18:29:27.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8003970 con 0x7ff2b4106a50 2024-03-28T18:29:27.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.892+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b419bf40 con 0x7ff2b4106a50 2024-03-28T18:29:27.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.893+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7ff2a8029070 con 0x7ff2b4106a50 2024-03-28T18:29:27.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.893+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b4102190 con 0x7ff2b4106a50 2024-03-28T18:29:27.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.893+0000 7ff2a37fe700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 0x7ff29c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:27.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.893+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff2a801e070 con 0x7ff2b4106a50 2024-03-28T18:29:27.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.893+0000 7ff2b1d9b700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 0x7ff29c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:27.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.894+0000 7ff2b1d9b700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 0x7ff29c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4006fd0 tx=0x7ff2a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:27.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:27.898+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff2a8013070 con 0x7ff2b4106a50 2024-03-28T18:29:28.097 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.097+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff2b41989d0 con 0x7ff2b4106a50 2024-03-28T18:29:28.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.182+0000 7ff2a37fe700 1 -- 172.21.15.145:0/2404666326 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff2a8030090 con 0x7ff2b4106a50 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 msgr2=0x7ff29c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 0x7ff29c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4006fd0 tx=0x7ff2a4006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 msgr2=0x7ff2b4197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8005850 tx=0x7ff2a8005030 comp rx=0 tx=0).stop 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 shutdown_connections 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff29c039020 0x7ff29c03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:28.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 --2- 172.21.15.145:0/2404666326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b4106a50 0x7ff2b4197670 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:28.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 >> 172.21.15.145:0/2404666326 conn(0x7ff2b40fc580 msgr2=0x7ff2b4104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:28.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 shutdown_connections 2024-03-28T18:29:28.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:28.185+0000 7ff2b8daf700 1 -- 172.21.15.145:0/2404666326 wait complete. 2024-03-28T18:29:28.186 INFO:teuthology.orchestra.run.smithi145.stderr:adjusted tunables profile to default 2024-03-28T18:29:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:28 smithi145 ceph-mon[32151]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:28 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:29:28.711 INFO:tasks.cephadm:Adding mon.a on smithi145 2024-03-28T18:29:28.711 INFO:tasks.cephadm:Adding mon.b on smithi164 2024-03-28T18:29:28.712 INFO:tasks.cephadm:Adding mon.c on smithi192 2024-03-28T18:29:28.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c' 2024-03-28T18:29:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-28T18:29:28.999 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-28T18:29:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:29:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:29.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:29.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:29.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:29.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:29 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:30.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:30.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:30.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 -- 172.21.15.192:0/2431747137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 msgr2=0x7fe8981021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 --2- 172.21.15.192:0/2431747137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981021c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe894009910 tx=0x7fe894009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 -- 172.21.15.192:0/2431747137 shutdown_connections 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 --2- 172.21.15.192:0/2431747137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981021c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 -- 172.21.15.192:0/2431747137 >> 172.21.15.192:0/2431747137 conn(0x7fe8980fd670 msgr2=0x7fe8980ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 -- 172.21.15.192:0/2431747137 shutdown_connections 2024-03-28T18:29:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.599+0000 7fe89fc09700 1 -- 172.21.15.192:0/2431747137 wait complete. 2024-03-28T18:29:30.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89fc09700 1 Processor -- start 2024-03-28T18:29:30.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89fc09700 1 -- start start 2024-03-28T18:29:30.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89fc09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:30.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89fc09700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89810c1b0 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89d9a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89d9a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.192:53916/0 (socket says 172.21.15.192:53916) 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.600+0000 7fe89d9a5700 1 -- 172.21.15.192:0/3517741566 learned_addr learned my addr 172.21.15.192:0/3517741566 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe89d9a5700 1 -- 172.21.15.192:0/3517741566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898197c20 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe89d9a5700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe894005850 tx=0x7fe894005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe894026070 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe898197f10 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89819bfb0 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe89400b6f0 con 0x7fe898101de0 2024-03-28T18:29:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.601+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe894030060 con 0x7fe898101de0 2024-03-28T18:29:30.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.602+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c005330 con 0x7fe898101de0 2024-03-28T18:29:30.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.602+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fe894029070 con 0x7fe898101de0 2024-03-28T18:29:30.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.602+0000 7fe88effd700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 0x7fe88403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:30.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.603+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe89401e070 con 0x7fe898101de0 2024-03-28T18:29:30.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.603+0000 7fe89d1a4700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 0x7fe88403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:30.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.603+0000 7fe89d1a4700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 0x7fe88403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe888006fd0 tx=0x7fe888006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:30.608 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.608+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe894013070 con 0x7fe898101de0 2024-03-28T18:29:30.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.813+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fe87c000c80 con 0x7fe884038fc0 2024-03-28T18:29:30.820 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.820+0000 7fe88effd700 1 -- 172.21.15.192:0/3517741566 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe87c000c80 con 0x7fe884038fc0 2024-03-28T18:29:30.820 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-03-28T18:29:30.822 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 msgr2=0x7fe88403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 0x7fe88403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe888006fd0 tx=0x7fe888006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 msgr2=0x7fe8981976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe894005850 tx=0x7fe894005030 comp rx=0 tx=0).stop 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 shutdown_connections 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe884038fc0 0x7fe88403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 --2- 172.21.15.192:0/3517741566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe898101de0 0x7fe8981976e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 >> 172.21.15.192:0/3517741566 conn(0x7fe8980fd670 msgr2=0x7fe8981037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 shutdown_connections 2024-03-28T18:29:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:30.822+0000 7fe89fc09700 1 -- 172.21.15.192:0/3517741566 wait complete. 2024-03-28T18:29:31.419 DEBUG:teuthology.orchestra.run.smithi164:mon.b> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T18:29:31.423 DEBUG:teuthology.orchestra.run.smithi192:mon.c> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T18:29:31.425 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:29:31.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:29:32.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:32.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: Saving service mon spec with placement smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c;count:3 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: Deploying daemon mon.c on smithi192 2024-03-28T18:29:32.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:31 smithi145 ceph-mon[32151]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:33.087 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:34.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:34 smithi192 systemd[1]: Starting Ceph mon.c for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:29:34.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:34 smithi145 ceph-mon[32151]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 -- 172.21.15.192:0/1432571166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 msgr2=0x7fca701021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 --2- 172.21.15.192:0/1432571166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca701021b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fca6800b0a0 tx=0x7fca6800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 -- 172.21.15.192:0/1432571166 shutdown_connections 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 --2- 172.21.15.192:0/1432571166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca701021b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 -- 172.21.15.192:0/1432571166 >> 172.21.15.192:0/1432571166 conn(0x7fca700fd660 msgr2=0x7fca700ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 -- 172.21.15.192:0/1432571166 shutdown_connections 2024-03-28T18:29:35.021 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.019+0000 7fca78010700 1 -- 172.21.15.192:0/1432571166 wait complete. 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca78010700 1 Processor -- start 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca78010700 1 -- start start 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca78010700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca78010700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7010c1a0 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca75dac700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca75dac700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.192:40918/0 (socket says 172.21.15.192:40918) 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca75dac700 1 -- 172.21.15.192:0/3596429136 learned_addr learned my addr 172.21.15.192:0/3596429136 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.020+0000 7fca75dac700 1 -- 172.21.15.192:0/3596429136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca70076380 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca75dac700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fca68015040 tx=0x7fca68003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca68017040 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca68022070 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca70074a30 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca6801d5d0 con 0x7fca70101dd0 2024-03-28T18:29:35.022 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.021+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca70074ef0 con 0x7fca70101dd0 2024-03-28T18:29:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.022+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fca6801d8d0 con 0x7fca70101dd0 2024-03-28T18:29:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.022+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca701032b0 con 0x7fca70101dd0 2024-03-28T18:29:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.022+0000 7fca66ffd700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 0x7fca5c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.022+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca6800f050 con 0x7fca70101dd0 2024-03-28T18:29:35.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.022+0000 7fca755ab700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 0x7fca5c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:35.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.023+0000 7fca755ab700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 0x7fca5c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca60006fd0 tx=0x7fca60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:35.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.027+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca68007e20 con 0x7fca70101dd0 2024-03-28T18:29:35.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 podman[36549]: 2024-03-28 18:29:35.110065399 +0000 UTC m=+0.184363491 container create b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:29:35.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 podman[36549]: 2024-03-28 18:29:35.023784773 +0000 UTC m=+0.098082873 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:29:35.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 podman[36549]: 2024-03-28 18:29:35.287370029 +0000 UTC m=+0.361668144 container init b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T18:29:35.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.311+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca70075a60 con 0x7fca70101dd0 2024-03-28T18:29:35.311 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.311+0000 7fca66ffd700 1 -- 172.21.15.192:0/3596429136 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7fca6801ce10 con 0x7fca70101dd0 2024-03-28T18:29:35.312 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:29:35.312 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","modified":"2024-03-28T18:27:56.935920Z","created":"2024-03-28T18:27:56.935920Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 msgr2=0x7fca5c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 0x7fca5c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca60006fd0 tx=0x7fca60006e40 comp rx=0 tx=0).stop 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 msgr2=0x7fca70074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fca68015040 tx=0x7fca68003d50 comp rx=0 tx=0).stop 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 shutdown_connections 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fca5c038fc0 0x7fca5c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 --2- 172.21.15.192:0/3596429136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca70101dd0 0x7fca70074490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:35.314 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 >> 172.21.15.192:0/3596429136 conn(0x7fca700fd660 msgr2=0x7fca701037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:35.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 shutdown_connections 2024-03-28T18:29:35.315 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:35.314+0000 7fca78010700 1 -- 172.21.15.192:0/3596429136 wait complete. 2024-03-28T18:29:35.315 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 podman[36549]: 2024-03-28 18:29:35.290802013 +0000 UTC m=+0.365100114 container start b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pidfile_write: ignore empty --pid-file 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: load: jerasure load: lrc 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: RocksDB version: 7.9.2 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Git sha 0 2024-03-28T18:29:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: DB SUMMARY 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: DB Session ID: HZLS7R3UKPJJOI46OB43 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: CURRENT file: CURRENT 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.error_if_exists: 0 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.create_if_missing: 0 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.paranoid_checks: 1 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T18:29:35.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.env: 0x5596365fcb20 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.info_log: 0x5596382a7920 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.statistics: (nil) 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.use_fsync: 0 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_log_file_size: 0 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T18:29:35.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_fallocate: 1 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.use_direct_reads: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.db_log_dir: 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.wal_dir: 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T18:29:35.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.write_buffer_manager: 0x55963827a1e0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T18:29:35.662 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.unordered_write: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.row_cache: None 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.wal_filter: None 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.two_write_queues: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.wal_compression: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.atomic_flush: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T18:29:35.663 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.log_readahead_size: 0 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_background_jobs: 2 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_background_compactions: -1 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_subcompactions: 1 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T18:29:35.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_open_files: -1 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_background_flushes: -1 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Compression algorithms supported: 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kZSTD supported: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kXpressCompression supported: 0 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T18:29:35.665 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kZlibCompression supported: 1 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kSnappyCompression supported: 1 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kLZ4Compression supported: 1 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: kBZip2Compression supported: 0 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.merge_operator: 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_filter: None 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T18:29:35.666 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5596382a7a20) 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_type: 0 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_index_type: 0 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_shortening: 1 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: checksum: 4 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: no_block_cache: 0 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache: 0x5596382ad090 2024-03-28T18:29:35.667 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_options: 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: capacity : 536870912 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_shard_bits : 4 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: strict_capacity_limit : 0 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_compressed: (nil) 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: persistent_cache: (nil) 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size: 4096 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size_deviation: 10 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_restart_interval: 16 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_block_restart_interval: 1 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: metadata_block_size: 4096 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: partition_filters: 0 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: use_delta_encoding: 1 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: filter_policy: bloomfilter 2024-03-28T18:29:35.668 INFO:journalctl@ceph.mon.c.smithi192.stdout: whole_key_filtering: 1 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: verify_compression: 0 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: format_version: 5 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_index_compression: 1 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_align: 0 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: max_auto_readahead_size: 262144 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: prepopulate_block_cache: 0 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: initial_auto_readahead_size: 8192 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression: NoCompression 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T18:29:35.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.num_levels: 7 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T18:29:35.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T18:29:35.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T18:29:35.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.table_properties_collectors: 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.inplace_update_support: 0 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.bloom_locality: 0 2024-03-28T18:29:35.673 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.max_successive_merges: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.ttl: 2592000 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enable_blob_files: false 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.min_blob_size: 0 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T18:29:35.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3667acbe-dc70-4514-9756-8edf04edee27 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650575322261, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650575322846, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711650575, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3667acbe-dc70-4514-9756-8edf04edee27", "db_session_id": "HZLS7R3UKPJJOI46OB43", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650575322984, "job": 1, "event": "recovery_finished"} 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5596383a0000 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: DB pointer 0x55963838a000 2024-03-28T18:29:35.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** DB Stats ** 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:29:35.676 INFO:journalctl@ceph.mon.c.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T18:29:35.677 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache BinnedLRUCache@0x5596382ad090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: using public_addr v2:172.21.15.192:0/0 -> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: starting mon.c rank -1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(???) e0 preinit fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).mds e1 new map 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: e1 2024-03-28T18:29:35.678 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout: legacy client fscid: -1 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout: No filesystems configured 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:35.679 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Deploying cephadm binary to smithi192 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Added host smithi192 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:29:35.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Saving service mon spec with placement smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c;count:3 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: Deploying daemon mon.c on smithi192 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 ceph-mon[36589]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 bash[36549]: b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 2024-03-28T18:29:35.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:35 smithi192 systemd[1]: Started Ceph mon.c for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:29:36.944 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:29:36.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:29:37.188 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: mon.a calling monitor election 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: mon.c calling monitor election 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: monmap e2: 2 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: fsmap 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: mgrmap e13: x(active, since 47s) 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: overall HEALTH_OK 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:40.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: mon.a calling monitor election 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: mon.c calling monitor election 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: monmap e2: 2 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: fsmap 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: mgrmap e13: x(active, since 47s) 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: overall HEALTH_OK 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:40.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:41.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.590+0000 7fd77bd71700 1 -- 172.21.15.192:0/4229960191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 msgr2=0x7fd760005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:41.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.590+0000 7fd77bd71700 1 --2- 172.21.15.192:0/4229960191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd760005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd774074460 tx=0x7fd77000feb0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.591+0000 7fd77bd71700 1 -- 172.21.15.192:0/4229960191 shutdown_connections 2024-03-28T18:29:41.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.591+0000 7fd77bd71700 1 --2- 172.21.15.192:0/4229960191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd760005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.591 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.591+0000 7fd77bd71700 1 -- 172.21.15.192:0/4229960191 >> 172.21.15.192:0/4229960191 conn(0x7fd7740fd760 msgr2=0x7fd7740ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:41.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.591+0000 7fd77bd71700 1 -- 172.21.15.192:0/4229960191 shutdown_connections 2024-03-28T18:29:41.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.591+0000 7fd77bd71700 1 -- 172.21.15.192:0/4229960191 wait complete. 2024-03-28T18:29:41.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 Processor -- start 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 -- start start 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd77419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410bf20 con 0x7fd7600035c0 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77bd71700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77410bda0 con 0x7fd77419fc80 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:47010/0 (socket says 172.21.15.192:47010) 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 -- 172.21.15.192:0/3561901124 learned_addr learned my addr 172.21.15.192:0/3561901124 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd779b0d700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd77419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 -- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 msgr2=0x7fd774199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 -- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 msgr2=0x7fd774199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.592+0000 7fd77930c700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:29:41.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd77930c700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd779b0d700 1 -- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 msgr2=0x7fd774199900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd779b0d700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd779b0d700 1 -- 172.21.15.192:0/3561901124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7741a02c0 con 0x7fd7600035c0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd779b0d700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd77419f740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd770010010 tx=0x7fd7700055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd770004b20 con 0x7fd7600035c0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd770013e40 con 0x7fd7600035c0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd774199e40 con 0x7fd7600035c0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd770011b60 con 0x7fd7600035c0 2024-03-28T18:29:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.593+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77419a3d0 con 0x7fd7600035c0 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.594+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fd770023070 con 0x7fd7600035c0 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.594+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd774075c70 con 0x7fd7600035c0 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.595+0000 7fd76affd700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 0x7fd75803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.595+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd770020050 con 0x7fd7600035c0 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.595+0000 7fd77930c700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 0x7fd75803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.595+0000 7fd77930c700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 0x7fd75803b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd764009180 tx=0x7fd764008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:41.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.600+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd770028020 con 0x7fd7600035c0 2024-03-28T18:29:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:41 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:41 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:41 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:41 smithi145 ceph-mon[32151]: Deploying daemon mon.b on smithi164 2024-03-28T18:29:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:41 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:41.883 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.883+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd77419ae30 con 0x7fd7600035c0 2024-03-28T18:29:41.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:41 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:41 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:41 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:41 smithi192 ceph-mon[36589]: Deploying daemon mon.b on smithi164 2024-03-28T18:29:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:41 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:41.884 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.884+0000 7fd76affd700 1 -- 172.21.15.192:0/3561901124 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7fd77001a550 con 0x7fd7600035c0 2024-03-28T18:29:41.884 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:29:41.884 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","modified":"2024-03-28T18:29:35.348926Z","created":"2024-03-28T18:27:56.935920Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:29:41.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 msgr2=0x7fd75803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:41.886 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 0x7fd75803b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd764009180 tx=0x7fd764008040 comp rx=0 tx=0).stop 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 msgr2=0x7fd77419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd77419f740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd770010010 tx=0x7fd7700055a0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 shutdown_connections 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd758038db0 0x7fd75803b270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7600035c0 0x7fd77419f740 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 --2- 172.21.15.192:0/3561901124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77419fc80 0x7fd774199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 >> 172.21.15.192:0/3561901124 conn(0x7fd7740fd760 msgr2=0x7fd7740ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 shutdown_connections 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:41.886+0000 7fd77bd71700 1 -- 172.21.15.192:0/3561901124 wait complete. 2024-03-28T18:29:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-03-28T18:29:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:42 smithi145 ceph-mon[32151]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:42 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/3561901124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:42 smithi192 ceph-mon[36589]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:42 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/3561901124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:43.489 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:29:43.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:29:43.729 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:29:44.216 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: RocksDB version: 7.9.2 2024-03-28T18:29:44.216 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Git sha 0 2024-03-28T18:29:44.216 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T18:29:44.216 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: DB SUMMARY 2024-03-28T18:29:44.216 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: DB Session ID: M7HFJYUTBCPTW3JOXUDJ 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: CURRENT file: CURRENT 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.error_if_exists: 0 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.create_if_missing: 0 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.paranoid_checks: 1 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.env: 0x55c3fba6bb20 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.info_log: 0x55c3fd04b920 2024-03-28T18:29:44.217 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.statistics: (nil) 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.use_fsync: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_log_file_size: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_fallocate: 1 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.use_direct_reads: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.db_log_dir: 2024-03-28T18:29:44.218 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.wal_dir: 2024-03-28T18:29:44.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T18:29:44.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T18:29:44.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T18:29:44.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.write_buffer_manager: 0x55c3fd01e1e0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.unordered_write: 0 2024-03-28T18:29:44.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.row_cache: None 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.wal_filter: None 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.two_write_queues: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.wal_compression: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.atomic_flush: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.log_readahead_size: 0 2024-03-28T18:29:44.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T18:29:44.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T18:29:44.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T18:29:44.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T18:29:44.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T18:29:44.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T18:29:44.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T18:29:44.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_background_jobs: 2 2024-03-28T18:29:44.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_background_compactions: -1 2024-03-28T18:29:44.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_subcompactions: 1 2024-03-28T18:29:44.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_open_files: -1 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_background_flushes: -1 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Compression algorithms supported: 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T18:29:44.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kZSTD supported: 0 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kXpressCompression supported: 0 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kZlibCompression supported: 1 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kSnappyCompression supported: 1 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kLZ4Compression supported: 1 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: kBZip2Compression supported: 0 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.merge_operator: 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_filter: None 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T18:29:44.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c3fd04ba20) 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: index_type: 0 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: data_block_index_type: 0 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: index_shortening: 1 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: checksum: 4 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: no_block_cache: 0 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_cache: 0x55c3fd051090 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_cache_options: 2024-03-28T18:29:44.719 INFO:journalctl@ceph.mon.b.smithi164.stdout: capacity : 536870912 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: num_shard_bits : 4 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: strict_capacity_limit : 0 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_cache_compressed: (nil) 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: persistent_cache: (nil) 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_size: 4096 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_size_deviation: 10 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_restart_interval: 16 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: index_block_restart_interval: 1 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: metadata_block_size: 4096 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: partition_filters: 0 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: use_delta_encoding: 1 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: filter_policy: bloomfilter 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: whole_key_filtering: 1 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: verify_compression: 0 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-03-28T18:29:44.720 INFO:journalctl@ceph.mon.b.smithi164.stdout: format_version: 5 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: enable_index_compression: 1 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: block_align: 0 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: max_auto_readahead_size: 262144 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: prepopulate_block_cache: 0 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: initial_auto_readahead_size: 8192 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression: NoCompression 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.num_levels: 7 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T18:29:44.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T18:29:44.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T18:29:44.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T18:29:44.724 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.table_properties_collectors: 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.inplace_update_support: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.bloom_locality: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.max_successive_merges: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.ttl: 2592000 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T18:29:44.725 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enable_blob_files: false 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.min_blob_size: 0 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 967de019-cdf1-44e1-9cee-1169b53c73fe 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650584217011, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650584217491, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711650584, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "967de019-cdf1-44e1-9cee-1169b53c73fe", "db_session_id": "M7HFJYUTBCPTW3JOXUDJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T18:29:44.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650584217573, "job": 1, "event": "recovery_finished"} 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c3fd144000 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: DB pointer 0x55c3fd12e000 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: ** DB Stats ** 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-28T18:29:44.727 INFO:journalctl@ceph.mon.b.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T18:29:44.728 INFO:journalctl@ceph.mon.b.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: Block cache BinnedLRUCache@0x55c3fd051090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: using public_addr v2:172.21.15.164:0/0 -> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: starting mon.b rank -1 at public addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] at bind addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(???) e0 preinit fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:44.729 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: e1 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: legacy client fscid: -1 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout: No filesystems configured 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T18:29:44.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:44.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Deploying cephadm binary to smithi192 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Added host smithi192 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.732 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2404666326' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:44.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Saving service mon spec with placement smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c;count:3 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Deploying daemon mon.c on smithi192 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.a calling monitor election 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.c calling monitor election 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: monmap e2: 2 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: fsmap 2024-03-28T18:29:44.735 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mgrmap e13: x(active, since 47s) 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: overall HEALTH_OK 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: Deploying daemon mon.b on smithi164 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/3561901124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 ceph-mon[36248]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 bash[36229]: a2f122f90cc3268b205ade63a08d50f2842cef9aea6da68c62a6384ca392e220 2024-03-28T18:29:44.736 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:44 smithi164 systemd[1]: Started Ceph mon.b for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:29:49.575 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: mon.c calling monitor election 2024-03-28T18:29:49.575 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: mon.a calling monitor election 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.576 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: fsmap 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: mgrmap e13: x(active, since 55s) 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: mon.b (rank 2) addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] is down (out of quorum) 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 -- 172.21.15.192:0/3237292432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a540035c0 msgr2=0x7f7a54005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 --2- 172.21.15.192:0/3237292432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a540035c0 0x7f7a54005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c074460 tx=0x7f7a6800fe60 comp rx=0 tx=0).stop 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 -- 172.21.15.192:0/3237292432 shutdown_connections 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 --2- 172.21.15.192:0/3237292432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a540035c0 0x7f7a54005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 -- 172.21.15.192:0/3237292432 >> 172.21.15.192:0/3237292432 conn(0x7f7a6c0fd730 msgr2=0x7f7a6c0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 -- 172.21.15.192:0/3237292432 shutdown_connections 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.574+0000 7f7a72e46700 1 -- 172.21.15.192:0/3237292432 wait complete. 2024-03-28T18:29:49.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.575+0000 7f7a72e46700 1 Processor -- start 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.575+0000 7f7a72e46700 1 -- start start 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.575+0000 7f7a72e46700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.575+0000 7f7a72e46700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a72e46700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 0x7f7a6c1a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a72e46700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c1102b0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a72e46700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c110130 con 0x7f7a6c19e490 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:47900/0 (socket says 172.21.15.192:47900) 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 -- 172.21.15.192:0/2214522808 learned_addr learned my addr 172.21.15.192:0/2214522808 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 0x7f7a6c1a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a63fff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c110430 con 0x7f7a540035c0 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 msgr2=0x7f7a6c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 msgr2=0x7f7a6c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:29:49.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a70be2700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a63fff700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 msgr2=0x7f7a6c1a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a63fff700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 msgr2=0x7f7a6c1a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a63fff700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a63fff700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 msgr2=0x7f7a6c19df50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 msgr2=0x7f7a6c1a27d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.576+0000 7f7a713e3700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c1aaa80 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a713e3700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 0x7f7a6c1a31c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400bd50 tx=0x7f7a6400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64019070 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c1aacb0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a6400cd10 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a640127e0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.577+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c1ab1a0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.578+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7a6401c070 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.578+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c0637c0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.578+0000 7f7a61ffb700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 0x7f7a5803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.578+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7a6404c8c0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.579+0000 7f7a70be2700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 0x7f7a5803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:49.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.579+0000 7f7a70be2700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 0x7f7a5803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7a68001eb0 tx=0x7f7a68005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:49.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.584+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a64013c30 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: mon.c calling monitor election 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: mon.a calling monitor election 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: fsmap 2024-03-28T18:29:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: mgrmap e13: x(active, since 55s) 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: mon.b (rank 2) addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] is down (out of quorum) 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:49.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.871+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a6c1ab4f0 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.871 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.871+0000 7f7a61ffb700 1 -- 172.21.15.192:0/2214522808 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7f7a6400e090 con 0x7f7a6c1a2d10 2024-03-28T18:29:49.871 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-28T18:29:49.871 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","modified":"2024-03-28T18:29:44.240268Z","created":"2024-03-28T18:27:56.935920Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 msgr2=0x7f7a5803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 0x7f7a5803b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7a68001eb0 tx=0x7f7a68005de0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 msgr2=0x7f7a6c1a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 0x7f7a6c1a31c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400bd50 tx=0x7f7a6400be30 comp rx=0 tx=0).stop 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 shutdown_connections 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a58039450 0x7f7a5803b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a6c1a2d10 0x7f7a6c1a31c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a540035c0 0x7f7a6c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 --2- 172.21.15.192:0/2214522808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a6c19e490 0x7f7a6c1a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:49.875 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.873+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 >> 172.21.15.192:0/2214522808 conn(0x7f7a6c0fd730 msgr2=0x7f7a6c196930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:49.875 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.874+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 shutdown_connections 2024-03-28T18:29:49.875 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:49.874+0000 7f7a72e46700 1 -- 172.21.15.192:0/2214522808 wait complete. 2024-03-28T18:29:49.875 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-03-28T18:29:50.474 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:50 smithi192 ceph-mon[36589]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:50.474 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:50.474 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:50 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/2214522808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:50.475 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:50.509 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T18:29:50.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T18:29:50.759 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:50 smithi145 ceph-mon[32151]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:50 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/2214522808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.c calling monitor election 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.a calling monitor election 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: fsmap 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mgrmap e13: x(active, since 55s) 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.b (rank 2) addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] is down (out of quorum) 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/2214522808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.b calling monitor election 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.b calling monitor election 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.a calling monitor election 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.c calling monitor election 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: fsmap 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: mgrmap e13: x(active, since 57s) 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: overall HEALTH_OK 2024-03-28T18:29:51.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:51 smithi164 ceph-mon[36248]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mon.b calling monitor election 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mon.b calling monitor election 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mon.a calling monitor election 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mon.c calling monitor election 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: fsmap 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: mgrmap e13: x(active, since 57s) 2024-03-28T18:29:51.876 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: overall HEALTH_OK 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.877 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:51 smithi145 ceph-mon[32151]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mon.b calling monitor election 2024-03-28T18:29:51.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mon.b calling monitor election 2024-03-28T18:29:51.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mon.a calling monitor election 2024-03-28T18:29:51.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mon.c calling monitor election 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: monmap e3: 3 mons at {a=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: fsmap 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: mgrmap e13: x(active, since 57s) 2024-03-28T18:29:51.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: overall HEALTH_OK 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi145:/etc/ceph/ceph.conf 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:51 smithi192 ceph-mon[36589]: Updating smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:29:52.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 -- 172.21.15.145:0/3161921013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1488106a40 msgr2=0x7f1488106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:52.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3161921013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1488106a40 0x7f1488106e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1478009910 tx=0x7f1478009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:52.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 -- 172.21.15.145:0/3161921013 shutdown_connections 2024-03-28T18:29:52.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3161921013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1488106a40 0x7f1488106e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 -- 172.21.15.145:0/3161921013 >> 172.21.15.145:0/3161921013 conn(0x7f1488077830 msgr2=0x7f14880febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:52.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 -- 172.21.15.145:0/3161921013 shutdown_connections 2024-03-28T18:29:52.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.313+0000 7f148cb2e700 1 -- 172.21.15.145:0/3161921013 wait complete. 2024-03-28T18:29:52.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 Processor -- start 2024-03-28T18:29:52.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 -- start start 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1488106a40 0x7f1488198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488198ca0 0x7f1488199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148810b030 con 0x7f14881a17d0 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148810aeb0 con 0x7f1488106a40 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148cb2e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148810b1b0 con 0x7f1488198ca0 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f1486d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f148659c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1488106a40 0x7f1488198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f1486d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:38606/0 (socket says 172.21.15.145:38606) 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f1486d9d700 1 -- 172.21.15.145:0/3208069108 learned_addr learned my addr 172.21.15.145:0/3208069108 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:29:52.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.314+0000 7f1485d9b700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488198ca0 0x7f1488199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 -- 172.21.15.145:0/3208069108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488198ca0 msgr2=0x7f1488199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488198ca0 0x7f1488199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 -- 172.21.15.145:0/3208069108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1488106a40 msgr2=0x7f1488198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1488106a40 0x7f1488198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 -- 172.21.15.145:0/3208069108 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14881a43d0 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f1486d9d700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f148000ba50 tx=0x7f148000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480019070 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f148000ce10 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14881a46c0 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480007350 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.315+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14881a4c00 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.316+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14880637c0 con 0x7f14881a17d0 2024-03-28T18:29:52.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.316+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f148001c070 con 0x7f14881a17d0 2024-03-28T18:29:52.322 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.317+0000 7f14777fe700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 0x7f147003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:52.322 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.317+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f148004b660 con 0x7f14881a17d0 2024-03-28T18:29:52.322 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.317+0000 7f148659c700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 0x7f147003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:52.322 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.317+0000 7f148659c700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 0x7f147003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1478009750 tx=0x7f147800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:52.322 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.322+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1480012500 con 0x7f14881a17d0 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.532+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f14881a4f20 con 0x7f14881a17d0 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.533+0000 7f14777fe700 1 -- 172.21.15.145:0/3208069108 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f1480014440 con 0x7f14881a17d0 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stdout:# minimal ceph.conf for b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:29:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: mon_host = [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 msgr2=0x7f147003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 0x7f147003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1478009750 tx=0x7f147800b540 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 msgr2=0x7f14881a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f148000ba50 tx=0x7f148000bd60 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 shutdown_connections 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1470039450 0x7f147003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14881a17d0 0x7f14881a3bc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1488106a40 0x7f1488198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 --2- 172.21.15.145:0/3208069108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1488198ca0 0x7f1488199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 >> 172.21.15.145:0/3208069108 conn(0x7f1488077830 msgr2=0x7f1488102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:52.536 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 shutdown_connections 2024-03-28T18:29:52.536 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:29:52.535+0000 7f148cb2e700 1 -- 172.21.15.145:0/3208069108 wait complete. 2024-03-28T18:29:53.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: Reconfiguring daemon mon.a on smithi145 2024-03-28T18:29:53.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:53.160 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:52 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3208069108' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: Reconfiguring daemon mon.a on smithi145 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:53.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:52 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3208069108' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:53.215 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:53.216 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.216 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: Reconfiguring daemon mon.a on smithi145 2024-03-28T18:29:53.216 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:29:53.216 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:52 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3208069108' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:53.243 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T18:29:53.243 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:53.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:29:53.274 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:53.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:53.341 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:29:53.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:29:53.377 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:29:53.377 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:53.447 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:29:53.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:29:53.482 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:29:53.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:29:53.552 INFO:tasks.cephadm:Adding mgr.x on smithi145 2024-03-28T18:29:53.552 INFO:tasks.cephadm:Adding mgr.y on smithi164 2024-03-28T18:29:53.552 INFO:tasks.cephadm:Adding mgr.z on smithi192 2024-03-28T18:29:53.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi145=x;smithi164=y;smithi192=z' 2024-03-28T18:29:53.823 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 -- 172.21.15.192:0/1458710291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 msgr2=0x7fc8f4103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/1458710291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4103810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e4009c20 comp rx=0 tx=0).stop 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 -- 172.21.15.192:0/1458710291 shutdown_connections 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/1458710291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4103810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 -- 172.21.15.192:0/1458710291 >> 172.21.15.192:0/1458710291 conn(0x7fc8f40fd2b0 msgr2=0x7fc8f40ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 -- 172.21.15.192:0/1458710291 shutdown_connections 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 -- 172.21.15.192:0/1458710291 wait complete. 2024-03-28T18:29:55.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.297+0000 7fc8f92c4700 1 Processor -- start 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 -- start start 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f4197b60 0x7fc8f419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4107e60 con 0x7fc8f41033f0 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4107ce0 con 0x7fc8f4197b60 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f92c4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4107fe0 con 0x7fc8f4197fe0 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:59834/0 (socket says 172.21.15.192:59834) 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 -- 172.21.15.192:0/4003365766 learned_addr learned my addr 172.21.15.192:0/4003365766 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f2ffd700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f27fc700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f4197b60 0x7fc8f419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 -- 172.21.15.192:0/4003365766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f4197b60 msgr2=0x7fc8f419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:55.301 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f4197b60 0x7fc8f419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 -- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 msgr2=0x7fc8f4197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 -- 172.21.15.192:0/4003365766 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f419fa30 con 0x7fc8f4197fe0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f2ffd700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8f37fe700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00ba50 tx=0x7fc8ec00be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.298+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec019070 con 0x7fc8f4197fe0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.299+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f419fd20 con 0x7fc8f4197fe0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.299+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8ec014440 con 0x7fc8f4197fe0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.299+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec007590 con 0x7fc8f4197fe0 2024-03-28T18:29:55.302 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.299+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f41a0280 con 0x7fc8f4197fe0 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.300+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc8ec01c070 con 0x7fc8f4197fe0 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.300+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40637c0 con 0x7fc8f4197fe0 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.300+0000 7fc8dbfff700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 0x7fc8dc03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.300+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc8ec04b650 con 0x7fc8f4197fe0 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.300+0000 7fc8f2ffd700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 0x7fc8dc03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:29:55.303 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.301+0000 7fc8f2ffd700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 0x7fc8dc03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b5c0 tx=0x7fc8e401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:29:55.306 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.306+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc8ec014910 con 0x7fc8f4197fe0 2024-03-28T18:29:55.516 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.515+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi145=x;smithi164=y;smithi192=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f4062720 con 0x7fc8dc039450 2024-03-28T18:29:55.523 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.522+0000 7fc8dbfff700 1 -- 172.21.15.192:0/4003365766 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc8f4062720 con 0x7fc8dc039450 2024-03-28T18:29:55.523 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-03-28T18:29:55.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 msgr2=0x7fc8dc03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:55.525 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 0x7fc8dc03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b5c0 tx=0x7fc8e401a040 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 msgr2=0x7fc8f419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00ba50 tx=0x7fc8ec00be10 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 shutdown_connections 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8dc039450 0x7fc8dc03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8f4197fe0 0x7fc8f419f160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8f41033f0 0x7fc8f4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 --2- 172.21.15.192:0/4003365766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f4197b60 0x7fc8f419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 >> 172.21.15.192:0/4003365766 conn(0x7fc8f40fd2b0 msgr2=0x7fc8f40fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 shutdown_connections 2024-03-28T18:29:55.526 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:29:55.524+0000 7fc8f92c4700 1 -- 172.21.15.192:0/4003365766 wait complete. 2024-03-28T18:29:55.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:55 smithi164 ceph-mon[36248]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:29:55.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:55 smithi164 ceph-mon[36248]: Reconfiguring daemon mon.b on smithi164 2024-03-28T18:29:55.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:55 smithi145 ceph-mon[32151]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:29:55.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:55 smithi145 ceph-mon[32151]: Reconfiguring daemon mon.b on smithi164 2024-03-28T18:29:55.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:55 smithi192 ceph-mon[36589]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:29:55.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:55 smithi192 ceph-mon[36589]: Reconfiguring daemon mon.b on smithi164 2024-03-28T18:29:56.121 DEBUG:teuthology.orchestra.run.smithi164:mgr.y> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T18:29:56.123 DEBUG:teuthology.orchestra.run.smithi192:mgr.z> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T18:29:56.125 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T18:29:56.126 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:29:56.126 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:29:56.145 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:29:56.145 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-03-28T18:29:56.202 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-28 18:29:02.998129348 +0000 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-28 18:23:53.609495340 +0000 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-28 18:23:53.609495340 +0000 2024-03-28T18:29:56.203 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-28 18:23:53.609495340 +0000 2024-03-28T18:29:56.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:29:56.272 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-28T18:29:56.272 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-28T18:29:56.272 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000202212 s, 2.5 MB/s 2024-03-28T18:29:56.274 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:29:56.331 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-03-28T18:29:56.386 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:29:56.386 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-28 18:29:03.023128798 +0000 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-28 18:23:53.952487470 +0000 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-28 18:23:53.952487470 +0000 2024-03-28T18:29:56.387 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-28 18:23:53.952487470 +0000 2024-03-28T18:29:56.387 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:29:56.454 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-28T18:29:56.454 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-28T18:29:56.454 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.00019587 s, 2.6 MB/s 2024-03-28T18:29:56.456 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:29:56.512 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-03-28T18:29:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:29:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:56.568 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-28T18:29:56.568 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:56.568 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:56.569 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-28 18:29:03.048128249 +0000 2024-03-28T18:29:56.569 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-28 18:23:54.281479920 +0000 2024-03-28T18:29:56.569 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-28 18:23:54.281479920 +0000 2024-03-28T18:29:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-28 18:23:54.281479920 +0000 2024-03-28T18:29:56.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi145=x;smithi164=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: Saving service mgr spec with placement smithi145=x;smithi164=y;smithi192=z;count:3 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:56.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:56.635 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-28T18:29:56.635 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-28T18:29:56.635 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000197449 s, 2.6 MB/s 2024-03-28T18:29:56.636 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:29:56.692 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-03-28T18:29:56.747 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:29:56.747 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-28 18:29:03.072127722 +0000 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-28 18:23:54.613472302 +0000 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-28 18:23:54.613472302 +0000 2024-03-28T18:29:56.748 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-28 18:23:54.613472302 +0000 2024-03-28T18:29:56.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi145=x;smithi164=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: Saving service mgr spec with placement smithi145=x;smithi164=y;smithi192=z;count:3 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:56.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:56.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:56.801 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-28T18:29:56.801 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-28T18:29:56.802 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.00016996 s, 3.0 MB/s 2024-03-28T18:29:56.803 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:29:56.858 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-28T18:29:56.858 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:29:56.876 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:29:56.876 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-28 18:29:43.698295544 +0000 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-28 18:24:11.279553247 +0000 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-28 18:24:11.279553247 +0000 2024-03-28T18:29:56.933 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-28 18:24:11.279553247 +0000 2024-03-28T18:29:56.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi145=x;smithi164=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: Saving service mgr spec with placement smithi145=x;smithi164=y;smithi192=z;count:3 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:29:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:29:56.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:56.991 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-28T18:29:56.991 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-28T18:29:56.991 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000288968 s, 1.8 MB/s 2024-03-28T18:29:56.992 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:29:57.050 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-28 18:29:43.699295527 +0000 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-28 18:24:11.663546540 +0000 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-28 18:24:11.663546540 +0000 2024-03-28T18:29:57.106 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-28 18:24:11.663546540 +0000 2024-03-28T18:29:57.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:29:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-28T18:29:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-28T18:29:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000229109 s, 2.2 MB/s 2024-03-28T18:29:57.174 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:29:57.231 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-28 18:29:43.699295527 +0000 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-28 18:24:12.008540515 +0000 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-28 18:24:12.008540515 +0000 2024-03-28T18:29:57.287 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-28 18:24:12.008540515 +0000 2024-03-28T18:29:57.287 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:29:57.352 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-28T18:29:57.353 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-28T18:29:57.353 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000195222 s, 2.6 MB/s 2024-03-28T18:29:57.354 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:29:57.410 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-03-28T18:29:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-28 18:29:43.700295511 +0000 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-28 18:24:12.341534698 +0000 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-28 18:24:12.341534698 +0000 2024-03-28T18:29:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-28 18:24:12.341534698 +0000 2024-03-28T18:29:57.465 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:29:57.530 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-28T18:29:57.530 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-28T18:29:57.530 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000193217 s, 2.6 MB/s 2024-03-28T18:29:57.531 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:29:57.587 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-28T18:29:57.587 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:29:57.602 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:29:57.602 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-03-28T18:29:57.657 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:29:57.657 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:57.657 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-28T18:29:57.657 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:57.658 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:57.658 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-28 18:29:34.508322776 +0000 2024-03-28T18:29:57.658 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-28 18:24:23.680818833 +0000 2024-03-28T18:29:57.658 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-28 18:24:23.680818833 +0000 2024-03-28T18:29:57.658 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-28 18:24:23.680818833 +0000 2024-03-28T18:29:57.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: Reconfiguring daemon mon.c on smithi192 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:29:57.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:29:57.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:29:57.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.724 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-28T18:29:57.724 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-28T18:29:57.724 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000228397 s, 2.2 MB/s 2024-03-28T18:29:57.725 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: Reconfiguring daemon mon.c on smithi192 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:29:57.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.781 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-28 18:29:34.509322757 +0000 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-28 18:24:24.006812280 +0000 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-28 18:24:24.006812280 +0000 2024-03-28T18:29:57.836 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-28 18:24:24.006812280 +0000 2024-03-28T18:29:57.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: Reconfiguring daemon mon.c on smithi192 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:29:57.887 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:29:57.900 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-28T18:29:57.901 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-28T18:29:57.901 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000172877 s, 3.0 MB/s 2024-03-28T18:29:57.902 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:29:57.957 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-03-28T18:29:58.012 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:29:58.012 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:58.012 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-28 18:29:34.509322757 +0000 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-28 18:24:24.323805908 +0000 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-28 18:24:24.323805908 +0000 2024-03-28T18:29:58.013 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-28 18:24:24.323805908 +0000 2024-03-28T18:29:58.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:29:58.080 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-28T18:29:58.080 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-28T18:29:58.080 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000177252 s, 2.9 MB/s 2024-03-28T18:29:58.081 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:29:58.136 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-03-28T18:29:58.191 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-28 18:29:34.510322738 +0000 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-28 18:24:24.652799295 +0000 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-28 18:24:24.652799295 +0000 2024-03-28T18:29:58.192 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-28 18:24:24.652799295 +0000 2024-03-28T18:29:58.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:29:58.257 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-28T18:29:58.257 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-28T18:29:58.257 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000195872 s, 2.6 MB/s 2024-03-28T18:29:58.259 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:29:58.314 INFO:tasks.cephadm:Deploying osd.0 on smithi145 with /dev/vg_nvme/lv_4... 2024-03-28T18:29:58.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:29:58.558 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:29:58.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:58 smithi145 ceph-mon[32151]: Deploying daemon mgr.z on smithi192 2024-03-28T18:29:58.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:29:58 smithi145 ceph-mon[32151]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:58.888 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:58 smithi192 ceph-mon[36589]: Deploying daemon mgr.z on smithi192 2024-03-28T18:29:58.888 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:29:58 smithi192 ceph-mon[36589]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:58 smithi164 ceph-mon[36248]: Deploying daemon mgr.z on smithi192 2024-03-28T18:29:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:29:58 smithi164 ceph-mon[36248]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:29:59.156 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 systemd[1]: Starting Ceph mgr.z for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:29:59.575 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 podman[38447]: 2024-03-28 18:29:59.177513185 +0000 UTC m=+0.017895502 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:29:59.575 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 podman[38447]: 2024-03-28 18:29:59.28199611 +0000 UTC m=+0.122378433 container create 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T18:29:59.575 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 podman[38447]: 2024-03-28 18:29:59.492560258 +0000 UTC m=+0.332942567 container init 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T18:29:59.575 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 podman[38447]: 2024-03-28 18:29:59.495191331 +0000 UTC m=+0.335573636 container start 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T18:29:59.575 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 ceph-mgr[38466]: -- 172.21.15.192:0/2905362830 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55799b4c05a0 con 0x55799b494c00 2024-03-28T18:29:59.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 bash[38447]: 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe 2024-03-28T18:29:59.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:29:59 smithi192 systemd[1]: Started Ceph mgr.z for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:30:00.324 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:30:00.325 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:30:00.325 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-28T18:30:00.325 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-28T18:30:00.325 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124436 s, 843 MB/s 2024-03-28T18:30:00.325 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-28T18:30:00.406 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:00.038+0000 7f83fa17d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:30:00.406 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:00.130+0000 7f83fa17d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:30:00.664 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:00.664 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.664 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.664 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.664 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:00.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:30:00.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:00.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:00.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:00 smithi145 ceph-mon[32151]: overall HEALTH_OK 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:00 smithi164 ceph-mon[36248]: overall HEALTH_OK 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:00.769+0000 7f83fa17d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:00.860+0000 7f83fa17d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:00.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:00 smithi192 ceph-mon[36589]: overall HEALTH_OK 2024-03-28T18:30:00.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_4 2024-03-28T18:30:01.156 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.002+0000 7f83fa17d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:30:01.157 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.097+0000 7f83fa17d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:30:01.167 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:30:01.441 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.180+0000 7f83fa17d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:30:01.441 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.441+0000 7f83fa17d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:30:01.576 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:01 smithi145 ceph-mon[32151]: Deploying daemon mgr.y on smithi164 2024-03-28T18:30:01.708 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:01 smithi164 ceph-mon[36248]: Deploying daemon mgr.y on smithi164 2024-03-28T18:30:01.728 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-mon[36589]: Deploying daemon mgr.y on smithi164 2024-03-28T18:30:01.728 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.551+0000 7f83fa17d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:30:01.728 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.635+0000 7f83fa17d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:30:01.728 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.727+0000 7f83fa17d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:30:01.961 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:01 smithi164 ceph-mgr[37632]: -- 172.21.15.164:0/2384643875 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b3bd5285a0 con 0x55b3bd4fd000 2024-03-28T18:30:02.156 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:01.903+0000 7f83fa17d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:30:02.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:01 smithi164 bash[37613]: 251d73a10f39be18c6ddba40cde2f49892a5b1ef20bb70a4bb8c1b22a691f05b 2024-03-28T18:30:02.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:01 smithi164 systemd[1]: Started Ceph mgr.y for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 -- 172.21.15.145:0/2002735904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 msgr2=0x7f3d301014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 --2- 172.21.15.145:0/2002735904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d301014d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800b0d0 tx=0x7f3d2800b3e0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 -- 172.21.15.145:0/2002735904 shutdown_connections 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 --2- 172.21.15.145:0/2002735904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30102c50 0x7f3d30109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 --2- 172.21.15.145:0/2002735904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d301014d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 --2- 172.21.15.145:0/2002735904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d301022b0 0x7f3d30102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.668+0000 7f3d3783d700 1 -- 172.21.15.145:0/2002735904 >> 172.21.15.145:0/2002735904 conn(0x7f3d300fc650 msgr2=0x7f3d300fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.669+0000 7f3d3783d700 1 -- 172.21.15.145:0/2002735904 shutdown_connections 2024-03-28T18:30:02.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.669+0000 7f3d3783d700 1 -- 172.21.15.145:0/2002735904 wait complete. 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.669+0000 7f3d3783d700 1 Processor -- start 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.669+0000 7f3d3783d700 1 -- start start 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 0x7f3d30197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102c50 0x7f3d3019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010ccf0 con 0x7f3d301010b0 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010cb70 con 0x7f3d301022b0 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d3783d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010ce70 con 0x7f3d30102c50 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d355d9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 0x7f3d30197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d35dda700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102c50 0x7f3d3019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d355d9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52864/0 (socket says 172.21.15.145:52864) 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d355d9700 1 -- 172.21.15.145:0/1603451244 learned_addr learned my addr 172.21.15.145:0/1603451244 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 -- 172.21.15.145:0/1603451244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102c50 msgr2=0x7f3d3019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102c50 0x7f3d3019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 -- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 msgr2=0x7f3d30197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 -- 172.21.15.145:0/1603451244 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3019ef90 con 0x7f3d301022b0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d355d9700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d34dd8700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 0x7f3d30197870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000d750 tx=0x7f3d2000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20014030 con 0x7f3d301022b0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.670+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d20004500 con 0x7f3d301022b0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.671+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3019f280 con 0x7f3d301022b0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.671+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20005480 con 0x7f3d301022b0 2024-03-28T18:30:02.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.671+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3006a960 con 0x7f3d301022b0 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.671+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30068e80 con 0x7f3d301022b0 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.671+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f3d2000f690 con 0x7f3d301022b0 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.672+0000 7f3d267fc700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 0x7f3d1c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.672+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d2004be30 con 0x7f3d301022b0 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.672+0000 7f3d355d9700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 0x7f3d1c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.672+0000 7f3d355d9700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 0x7f3d1c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3d28007fa0 tx=0x7f3d2800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:02.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.681+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d30068e80 con 0x7f3d301022b0 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:02.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:02.892+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d301082e0 con 0x7f3d1c039200 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:02.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:02.963 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:02.482+0000 7f98552c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:30:02.963 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:02.579+0000 7f98552c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:30:03.355 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.259+0000 7f98552c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:30:03.355 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.353+0000 7f98552c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:30:03.406 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:02.935+0000 7f83fa17d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:30:03.406 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:03.070+0000 7f83fa17d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:30:03.686 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.502+0000 7f98552c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.601+0000 7f98552c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:30:03.687 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.685+0000 7f98552c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:30:03.724 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:03.427+0000 7f83fa17d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:03.525+0000 7f83fa17d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:30:03.725 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:03.724+0000 7f83fa17d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:30:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:03.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:03.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.157 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:03.908+0000 7f83fa17d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:30:04.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:03.952+0000 7f98552c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:30:04.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:04.062+0000 7f98552c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:30:04.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:04.148+0000 7f98552c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:30:04.449 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:04.189+0000 7f83fa17d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:30:04.450 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:04.449+0000 7f83fa17d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:30:04.540 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:04.243+0000 7f98552c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:30:04.540 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:04.418+0000 7f98552c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:04.645 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:04 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:04.906 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:04.684+0000 7f83fa17d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:04 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:30:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:04 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:05.712 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:05.387+0000 7f98552c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:30:05.712 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:05.517+0000 7f98552c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:30:05.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:30:05.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: Reconfiguring daemon mgr.x on smithi145 2024-03-28T18:30:05.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]: dispatch 2024-03-28T18:30:05.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]': finished 2024-03-28T18:30:05.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:30:05.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:05.481+0000 7f83fa17d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: Reconfiguring daemon mgr.x on smithi145 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]: dispatch 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]': finished 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:30:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: Reconfiguring daemon mgr.x on smithi145 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]: dispatch 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2932258233' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d20b632f-9b4a-4f94-82e3-3b5668acd7f0"}]': finished 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:30:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:06.149 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:05.861+0000 7f98552c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:30:06.149 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:05.951+0000 7f98552c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:30:06.149 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:06.148+0000 7f98552c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:30:06.359 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:06.050+0000 7f83fa17d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:30:06.359 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:06.146+0000 7f83fa17d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:30:06.359 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z[38462]: 2024-03-28T18:30:06.359+0000 7f83fa17d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:30:06.462 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:06.333+0000 7f98552c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:30:06.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:06.551+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f3d20004750 con 0x7f3d301022b0 2024-03-28T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3586150969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: Standby manager daemon z started 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:30:06.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:06 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:06.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:06.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3586150969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:06.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: Standby manager daemon z started 2024-03-28T18:30:06.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:30:06.874 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:06.874 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:30:06.874 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:06.874 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:06.614+0000 7f98552c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:30:06.874 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:06.872+0000 7f98552c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3586150969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: Standby manager daemon z started 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:30:07.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:06 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.192:0/44667951' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:07.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:07.106+0000 7f98552c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:30:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:07 smithi164 ceph-mon[36248]: mgrmap e14: x(active, since 73s), standbys: z 2024-03-28T18:30:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:07 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:30:07.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:07 smithi192 ceph-mon[36589]: mgrmap e14: x(active, since 73s), standbys: z 2024-03-28T18:30:07.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:07 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:30:08.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:07 smithi145 ceph-mon[32151]: mgrmap e14: x(active, since 73s), standbys: z 2024-03-28T18:30:08.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:07 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:30:08.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:07.902+0000 7f98552c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:30:08.782 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:08 smithi164 ceph-mon[36248]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:08.782 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:08.469+0000 7f98552c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:30:08.782 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:08.567+0000 7f98552c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:30:08.782 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:30:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y[37628]: 2024-03-28T18:30:08.782+0000 7f98552c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:30:08.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:08 smithi192 ceph-mon[36589]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:09.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:08 smithi145 ceph-mon[32151]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:09.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:09.572+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mon.2 v2:172.21.15.164:3300/0 8 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f3d20020350 con 0x7f3d301022b0 2024-03-28T18:30:09.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:09 smithi145 ceph-mon[32151]: Standby manager daemon y started 2024-03-28T18:30:09.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:09 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:30:09.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:09 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:09.646 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:09 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:30:09.647 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:09 smithi145 ceph-mon[32151]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:09 smithi192 ceph-mon[36589]: Standby manager daemon y started 2024-03-28T18:30:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:09 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:30:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:09 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:09 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:30:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:09 smithi192 ceph-mon[36589]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:09 smithi164 ceph-mon[36248]: Standby manager daemon y started 2024-03-28T18:30:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:09 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:30:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:09 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:30:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:09 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:30:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:09 smithi164 ceph-mon[36248]: from='mgr.? 172.21.15.164:0/4152662834' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:30:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:10 smithi192 ceph-mon[36589]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:10 smithi192 ceph-mon[36589]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-03-28T18:30:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:30:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:10 smithi164 ceph-mon[36248]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:10 smithi164 ceph-mon[36248]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-03-28T18:30:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:30:11.011 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:10 smithi145 ceph-mon[32151]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:11.011 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:10 smithi145 ceph-mon[32151]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-03-28T18:30:11.011 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:11.900 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:11 smithi145 ceph-mon[32151]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:11 smithi192 ceph-mon[36589]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:11 smithi164 ceph-mon[36248]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:30:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:13.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:30:13.515 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:13.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:30:13.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:14 smithi164 ceph-mon[36248]: Deploying daemon osd.0 on smithi145 2024-03-28T18:30:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:14 smithi164 ceph-mon[36248]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:14.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:14 smithi145 ceph-mon[32151]: Deploying daemon osd.0 on smithi145 2024-03-28T18:30:14.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:14 smithi145 ceph-mon[32151]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:14.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:14 smithi192 ceph-mon[36589]: Deploying daemon osd.0 on smithi145 2024-03-28T18:30:14.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:14 smithi192 ceph-mon[36589]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:16.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:16 smithi145 ceph-mon[32151]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:16.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:16 smithi192 ceph-mon[36589]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:16 smithi164 ceph-mon[36248]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:18.764 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:18 smithi145 ceph-mon[32151]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:18.764 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.764 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.764 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:18 smithi192 ceph-mon[36589]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:18 smithi164 ceph-mon[36248]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:20.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:20 smithi145 ceph-mon[32151]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:20 smithi192 ceph-mon[36589]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:20.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:20 smithi164 ceph-mon[36248]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:21.901 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 0 on host 'smithi145' 2024-03-28T18:30:21.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.900+0000 7f3d267fc700 1 -- 172.21.15.145:0/1603451244 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3d301082e0 con 0x7f3d1c039200 2024-03-28T18:30:21.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.902+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 msgr2=0x7f3d1c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.902+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 0x7f3d1c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3d28007fa0 tx=0x7f3d2800b040 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.902+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 msgr2=0x7f3d30197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.902+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 0x7f3d30197870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000d750 tx=0x7f3d2000da60 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 shutdown_connections 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d1c039200 0x7f3d1c03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102c50 0x7f3d3019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301010b0 0x7f3d30197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 --2- 172.21.15.145:0/1603451244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d301022b0 0x7f3d30197870 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 >> 172.21.15.145:0/1603451244 conn(0x7f3d300fc650 msgr2=0x7f3d30105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 shutdown_connections 2024-03-28T18:30:21.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:30:21.903+0000 7f3d3783d700 1 -- 172.21.15.145:0/1603451244 wait complete. 2024-03-28T18:30:22.197 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.197 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:22.198 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:22.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:22.499 DEBUG:teuthology.orchestra.run.smithi145:osd.0> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T18:30:22.502 INFO:tasks.cephadm:Deploying osd.1 on smithi164 with /dev/vg_nvme/lv_4... 2024-03-28T18:30:22.502 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:30:22.746 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:30:23.192 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:23 smithi164 ceph-mon[36248]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:30:23.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:23 smithi192 ceph-mon[36589]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:30:23.429 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:23 smithi145 ceph-mon[32151]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:30:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:24 smithi192 ceph-mon[36589]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:30:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:24 smithi192 ceph-mon[36589]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:30:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:24 smithi192 ceph-mon[36589]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:30:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:24 smithi192 ceph-mon[36589]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:24 smithi164 ceph-mon[36248]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:30:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:24 smithi164 ceph-mon[36248]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:30:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:24 smithi164 ceph-mon[36248]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:30:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:24 smithi164 ceph-mon[36248]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:24.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-mon[32151]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:30:24.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-mon[32151]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:30:24.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-mon[32151]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:30:24.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:24.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-mon[32151]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:24.527 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0[43962]: 2024-03-28T18:30:24.119+0000 7fe6b1745700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T18:30:24.527 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:30:24 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0[43962]: 2024-03-28T18:30:24.122+0000 7fe6ad538700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130898 s, 801 MB/s 2024-03-28T18:30:24.609 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-28T18:30:25.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:25 smithi164 ceph-mon[36248]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:30:25.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:25 smithi164 ceph-mon[36248]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:30:25.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:25.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:25.188 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_4 2024-03-28T18:30:25.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:25 smithi192 ceph-mon[36589]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:30:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:25 smithi192 ceph-mon[36589]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:30:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:25.434 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:30:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:25 smithi145 ceph-mon[32151]: from='osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:30:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:25 smithi145 ceph-mon[32151]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:30:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.382 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:30:26.382 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] boot 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:26.383 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] boot 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:26.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:26.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:30:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:30:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: osd.0 [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] boot 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- 172.21.15.164:0/2408477757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 msgr2=0x7f0234103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 --2- 172.21.15.164:0/2408477757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f0234103840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f02240099c0 tx=0x7f0224009cd0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- 172.21.15.164:0/2408477757 shutdown_connections 2024-03-28T18:30:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 --2- 172.21.15.164:0/2408477757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 0x7f023410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 --2- 172.21.15.164:0/2408477757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 0x7f0234102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 --2- 172.21.15.164:0/2408477757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f0234103840 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- 172.21.15.164:0/2408477757 >> 172.21.15.164:0/2408477757 conn(0x7f02340fd7a0 msgr2=0x7f02340ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- 172.21.15.164:0/2408477757 shutdown_connections 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- 172.21.15.164:0/2408477757 wait complete. 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 Processor -- start 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.854+0000 7f023b696700 1 -- start start 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 0x7f0234198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 0x7f023419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023410dda0 con 0x7f02341021e0 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023410dc20 con 0x7f02341033e0 2024-03-28T18:30:26.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023410df20 con 0x7f0234103d80 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49680/0 (socket says 172.21.15.164:49680) 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 -- 172.21.15.164:0/2814792677 learned_addr learned my addr 172.21.15.164:0/2814792677 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0239c33700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 0x7f023419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0239432700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 0x7f0234198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 -- 172.21.15.164:0/2814792677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 msgr2=0x7f023419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 0x7f023419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 -- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 msgr2=0x7f0234198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 0x7f0234198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 -- 172.21.15.164:0/2814792677 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023419fef0 con 0x7f02341033e0 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f0238c31700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0224000c00 tx=0x7f02240057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:26.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224027070 con 0x7f02341033e0 2024-03-28T18:30:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02341a0180 con 0x7f02341033e0 2024-03-28T18:30:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0224004150 con 0x7f02341033e0 2024-03-28T18:30:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400f460 con 0x7f02341033e0 2024-03-28T18:30:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.855+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02341a05e0 con 0x7f02341033e0 2024-03-28T18:30:26.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.856+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0234068e50 con 0x7f02341033e0 2024-03-28T18:30:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.857+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f022402a070 con 0x7f02341033e0 2024-03-28T18:30:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.858+0000 7f022a7fc700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 0x7f02200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.858+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f02240cde50 con 0x7f02341033e0 2024-03-28T18:30:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.858+0000 7f0239432700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 0x7f02200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:26.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.859+0000 7f0239432700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 0x7f02200a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0230005fd0 tx=0x7f0230005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:26.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:26.862+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0224013070 con 0x7f02341033e0 2024-03-28T18:30:27.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:27.068+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f02340625d0 con 0x7f02200a2800 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi145 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi145 to 17088M 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi145 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi145 to 17088M 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi145 2024-03-28T18:30:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi145 to 17088M 2024-03-28T18:30:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:30:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:28.341 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:28 smithi164 ceph-mon[36248]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:28.342 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:28 smithi164 ceph-mon[36248]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:28.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:28 smithi192 ceph-mon[36589]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:28.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:28 smithi192 ceph-mon[36589]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:28 smithi145 ceph-mon[32151]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:28 smithi145 ceph-mon[32151]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/1982113451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]': finished 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:30 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/474604453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/1982113451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]': finished 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:30 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/474604453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/1982113451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]: dispatch 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70cfca66-0490-4c8f-9d54-0a64f05392a0"}]': finished 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:30 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/474604453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:32.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:32 smithi192 ceph-mon[36589]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:32.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:32 smithi145 ceph-mon[32151]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:32 smithi164 ceph-mon[36248]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:34.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:34 smithi164 ceph-mon[36248]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:34.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:34 smithi145 ceph-mon[32151]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:34.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:34 smithi192 ceph-mon[36589]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:36.737 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:36 smithi164 ceph-mon[36248]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:36 smithi145 ceph-mon[32151]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:36 smithi192 ceph-mon[36589]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:37.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:30:37.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:37.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:30:37.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:30:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:38.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:38 smithi145 ceph-mon[32151]: Deploying daemon osd.1 on smithi164 2024-03-28T18:30:38.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:38 smithi145 ceph-mon[32151]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:38.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:38 smithi164 ceph-mon[36248]: Deploying daemon osd.1 on smithi164 2024-03-28T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:38 smithi164 ceph-mon[36248]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:38 smithi192 ceph-mon[36589]: Deploying daemon osd.1 on smithi164 2024-03-28T18:30:38.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:38 smithi192 ceph-mon[36589]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:40.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:40 smithi164 ceph-mon[36248]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:40.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:40 smithi145 ceph-mon[32151]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:40.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:40 smithi192 ceph-mon[36589]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:42.768 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:42 smithi164 ceph-mon[36248]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:42.768 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:42 smithi145 ceph-mon[32151]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:42.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:42 smithi192 ceph-mon[36589]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:44 smithi145 ceph-mon[32151]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:44 smithi192 ceph-mon[36589]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:44.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:44 smithi164 ceph-mon[36248]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:45.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.983+0000 7f022a7fc700 1 -- 172.21.15.164:0/2814792677 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f02340625d0 con 0x7f02200a2800 2024-03-28T18:30:45.983 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 1 on host 'smithi164' 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 msgr2=0x7f02200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 0x7f02200a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0230005fd0 tx=0x7f0230005e50 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 msgr2=0x7f02341987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0224000c00 tx=0x7f02240057c0 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 shutdown_connections 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02200a2800 0x7f02200a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0234103d80 0x7f023419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02341021e0 0x7f0234198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 --2- 172.21.15.164:0/2814792677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02341033e0 0x7f02341987d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.985+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 >> 172.21.15.164:0/2814792677 conn(0x7f02340fd7a0 msgr2=0x7f0234106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.986+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 shutdown_connections 2024-03-28T18:30:45.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:30:45.986+0000 7f023b696700 1 -- 172.21.15.164:0/2814792677 wait complete. 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:46.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:46.630 DEBUG:teuthology.orchestra.run.smithi164:osd.1> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T18:30:46.632 INFO:tasks.cephadm:Deploying osd.2 on smithi192 with /dev/vg_nvme/lv_4... 2024-03-28T18:30:46.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:30:46.875 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:30:47.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:46 smithi164 ceph-mon[36248]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:47.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:46 smithi164 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:47.254 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:46 smithi192 ceph-mon[36589]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:47.254 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:46 smithi192 ceph-mon[36589]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:47.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:46 smithi145 ceph-mon[32151]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:47.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:46 smithi145 ceph-mon[32151]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.141 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:47 smithi192 ceph-mon[36589]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:47 smithi164 ceph-mon[36248]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:48.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:30:48.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: from='osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:30:48.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:48.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:30:48.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:47 smithi145 ceph-mon[32151]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:48.678 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:30:48.679 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:30:48.679 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-28T18:30:48.679 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-28T18:30:48.679 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113491 s, 924 MB/s 2024-03-28T18:30:48.679 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-28T18:30:48.712 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1[40830]: 2024-03-28T18:30:48.514+0000 7f3cf6d63700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T18:30:48.713 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1[40830]: 2024-03-28T18:30:48.518+0000 7f3ceeb4e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:30:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:30:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-mon[36248]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:30:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:48 smithi164 ceph-mon[36248]: from='osd.1 ' entity='osd.1' 2024-03-28T18:30:49.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-03-28T18:30:49.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:48 smithi192 ceph-mon[36589]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:30:49.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:48 smithi192 ceph-mon[36589]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:30:49.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:48 smithi192 ceph-mon[36589]: from='osd.1 ' entity='osd.1' 2024-03-28T18:30:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:48 smithi145 ceph-mon[32151]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:30:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:48 smithi145 ceph-mon[32151]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:30:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:48 smithi145 ceph-mon[32151]: from='osd.1 ' entity='osd.1' 2024-03-28T18:30:49.456 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:30:50.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] boot 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi164 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:50.163 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] boot 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi164 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:50.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: osd.1 [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] boot 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:30:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi164 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:30:50.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 -- 172.21.15.192:0/738689554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec100860 msgr2=0x7ff1ec100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/738689554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec100860 0x7ff1ec100c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009990 tx=0x7ff1dc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 -- 172.21.15.192:0/738689554 shutdown_connections 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/738689554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec102400 0x7ff1ec108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/738689554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec100860 0x7ff1ec100c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/738689554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec101a60 0x7ff1ec101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 -- 172.21.15.192:0/738689554 >> 172.21.15.192:0/738689554 conn(0x7ff1ec0fbe20 msgr2=0x7ff1ec0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:30:50.954 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 -- 172.21.15.192:0/738689554 shutdown_connections 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 -- 172.21.15.192:0/738689554 wait complete. 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.951+0000 7ff1f22fa700 1 Processor -- start 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 -- start start 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec100860 0x7ff1ec10a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 0x7ff1ec19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec10cd20 con 0x7ff1ec102400 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec10cba0 con 0x7ff1ec100860 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f22fa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec10cea0 con 0x7ff1ec101a60 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:45420/0 (socket says 172.21.15.192:45420) 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 -- 172.21.15.192:0/616998319 learned_addr learned my addr 172.21.15.192:0/616998319 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:30:50.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1ebfff700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec100860 0x7ff1ec10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f0897700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 0x7ff1ec19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 -- 172.21.15.192:0/616998319 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec100860 msgr2=0x7ff1ec10a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec100860 0x7ff1ec10a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 -- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 msgr2=0x7ff1ec19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 0x7ff1ec19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 -- 172.21.15.192:0/616998319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec1a0030 con 0x7ff1ec101a60 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1f0897700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 0x7ff1ec19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.952+0000 7ff1eb7fe700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b530 tx=0x7ff1e000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.953+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0015020 con 0x7ff1ec101a60 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.953+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec1a0320 con 0x7ff1ec101a60 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.953+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1e0011d10 con 0x7ff1ec101a60 2024-03-28T18:30:50.956 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.953+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000f2e0 con 0x7ff1ec101a60 2024-03-28T18:30:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.953+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec1a08e0 con 0x7ff1ec101a60 2024-03-28T18:30:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.954+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec068e50 con 0x7ff1ec101a60 2024-03-28T18:30:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.955+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7ff1e0018020 con 0x7ff1ec101a60 2024-03-28T18:30:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.955+0000 7ff1e97fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 0x7ff1d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:30:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.955+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7ff1e00cca20 con 0x7ff1ec101a60 2024-03-28T18:30:50.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.956+0000 7ff1ebfff700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 0x7ff1d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:30:50.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.956+0000 7ff1ebfff700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 0x7ff1d40a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000c00 tx=0x7ff1dc003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:30:50.960 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:50.960+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1e001f020 con 0x7ff1ec101a60 2024-03-28T18:30:51.165 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:30:51.164+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1ec0625d0 con 0x7ff1d40a2800 2024-03-28T18:30:51.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:50 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi164 to 17080M 2024-03-28T18:30:51.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:50 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi164 to 17080M 2024-03-28T18:30:51.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:50 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi164 to 17080M 2024-03-28T18:30:52.018 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:30:52.018 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:52.019 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:52.019 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:52.019 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:52.019 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:52 smithi192 ceph-mon[36589]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:52.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:30:52.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:52.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:52.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:52.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:52.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:52 smithi145 ceph-mon[32151]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:30:52.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:52 smithi164 ceph-mon[36248]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/3830484370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]': finished 2024-03-28T18:30:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:30:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:30:54.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:54 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/1389681575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/3830484370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]': finished 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:30:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:30:54.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:54 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/1389681575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/3830484370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]: dispatch 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36e9e51d-79a5-402a-bf8b-9bb427e8e847"}]': finished 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:30:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:54 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/1389681575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:30:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:56 smithi145 ceph-mon[32151]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:56.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:56 smithi192 ceph-mon[36589]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:56 smithi164 ceph-mon[36248]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:58.759 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:30:58 smithi192 ceph-mon[36589]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:30:58 smithi145 ceph-mon[32151]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:30:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:30:58 smithi164 ceph-mon[36248]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:00 smithi145 ceph-mon[32151]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:00 smithi192 ceph-mon[36589]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:00 smithi164 ceph-mon[36248]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:01.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:31:01.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:01.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:31:01.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:01.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:31:01.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:02 smithi145 ceph-mon[32151]: Deploying daemon osd.2 on smithi192 2024-03-28T18:31:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:02 smithi145 ceph-mon[32151]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:02.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:02 smithi192 ceph-mon[36589]: Deploying daemon osd.2 on smithi192 2024-03-28T18:31:02.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:02 smithi192 ceph-mon[36589]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:02 smithi164 ceph-mon[36248]: Deploying daemon osd.2 on smithi192 2024-03-28T18:31:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:02 smithi164 ceph-mon[36248]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:04 smithi145 ceph-mon[32151]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:04 smithi192 ceph-mon[36589]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:04 smithi164 ceph-mon[36248]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:06 smithi145 ceph-mon[32151]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:06 smithi192 ceph-mon[36589]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:06 smithi164 ceph-mon[36248]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:08.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:08 smithi192 ceph-mon[36589]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:08 smithi145 ceph-mon[32151]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:08 smithi164 ceph-mon[36248]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:09.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.278+0000 7ff1e97fa700 1 -- 172.21.15.192:0/616998319 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff1ec0625d0 con 0x7ff1d40a2800 2024-03-28T18:31:09.279 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 2 on host 'smithi192' 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.280+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 msgr2=0x7ff1d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.280+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 0x7ff1d40a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000c00 tx=0x7ff1dc003960 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.280+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 msgr2=0x7ff1ec19d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.280+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b530 tx=0x7ff1e000b840 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 shutdown_connections 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1d40a2800 0x7ff1d40a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ec102400 0x7ff1ec19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1ec100860 0x7ff1ec10a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 --2- 172.21.15.192:0/616998319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1ec101a60 0x7ff1ec19d0d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 >> 172.21.15.192:0/616998319 conn(0x7ff1ec0fbe20 msgr2=0x7ff1ec104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:31:09.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 shutdown_connections 2024-03-28T18:31:09.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:31:09.281+0000 7ff1f22fa700 1 -- 172.21.15.192:0/616998319 wait complete. 2024-03-28T18:31:09.904 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.904 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.904 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.931 DEBUG:teuthology.orchestra.run.smithi192:osd.2> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:09.975 INFO:tasks.cephadm:Deploying osd.3 on smithi145 with /dev/vg_nvme/lv_3... 2024-03-28T18:31:09.975 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:10.005 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:10.221 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:31:10.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:10 smithi145 ceph-mon[32151]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:10.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:10 smithi145 ceph-mon[32151]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:10.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:10 smithi145 ceph-mon[32151]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:10 smithi192 ceph-mon[36589]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:10 smithi192 ceph-mon[36589]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:10 smithi192 ceph-mon[36589]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:10 smithi164 ceph-mon[36248]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:10 smithi164 ceph-mon[36248]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:10 smithi164 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:31:11.642 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:11 smithi145 ceph-mon[32151]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:31:11.642 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:11 smithi145 ceph-mon[32151]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:31:11.642 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:11 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:11.642 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:11 smithi145 ceph-mon[32151]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:11.643 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:11 smithi145 ceph-mon[32151]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:11.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-mon[36589]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:31:11.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-mon[36589]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:31:11.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:11.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-mon[36589]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:11.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-mon[36589]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:11.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2[41773]: 2024-03-28T18:31:11.552+0000 7f3103cc4700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T18:31:11.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2[41773]: 2024-03-28T18:31:11.557+0000 7f30ffab7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:31:11.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:11 smithi164 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:31:11.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:11 smithi164 ceph-mon[36248]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:31:11.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:11 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:11.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:11 smithi164 ceph-mon[36248]: from='osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:11.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:11 smithi164 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129105 s, 812 MB/s 2024-03-28T18:31:12.215 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-28T18:31:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:12 smithi145 ceph-mon[32151]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:12 smithi145 ceph-mon[32151]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:31:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:12 smithi145 ceph-mon[32151]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:31:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:12.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:12.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:12 smithi192 ceph-mon[36589]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:12.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:12 smithi192 ceph-mon[36589]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:31:12.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:12 smithi192 ceph-mon[36589]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:31:12.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:12.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:12.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_3 2024-03-28T18:31:12.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:12 smithi164 ceph-mon[36248]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:12.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:12 smithi164 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:31:12.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:12 smithi164 ceph-mon[36248]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:31:12.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:12.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:13.136 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:31:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:31:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] boot 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:13.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] boot 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:31:14.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: osd.2 [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] boot 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:14.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- 172.21.15.145:0/2093690117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98102400 msgr2=0x7efd98108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2093690117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98102400 0x7efd98108cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efd9000b0d0 tx=0x7efd9000b3e0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- 172.21.15.145:0/2093690117 shutdown_connections 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2093690117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98102400 0x7efd98108cb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2093690117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98100860 0x7efd98100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2093690117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98101a60 0x7efd98101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- 172.21.15.145:0/2093690117 >> 172.21.15.145:0/2093690117 conn(0x7efd980fbde0 msgr2=0x7efd980fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- 172.21.15.145:0/2093690117 shutdown_connections 2024-03-28T18:31:14.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- 172.21.15.145:0/2093690117 wait complete. 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 Processor -- start 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 -- start start 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98100860 0x7efd981037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.711+0000 7efd9fcac700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98102400 0x7efd9819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9fcac700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9810c450 con 0x7efd98101a60 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9fcac700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9810c2d0 con 0x7efd98102400 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9fcac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9810c5d0 con 0x7efd98100860 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:42354/0 (socket says 172.21.15.145:42354) 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 -- 172.21.15.145:0/2358499655 learned_addr learned my addr 172.21.15.145:0/2358499655 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9da48700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98100860 0x7efd981037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9e249700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98102400 0x7efd9819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:14.714 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 -- 172.21.15.145:0/2358499655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98100860 msgr2=0x7efd981037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98100860 0x7efd981037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 -- 172.21.15.145:0/2358499655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98102400 msgr2=0x7efd9819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98102400 0x7efd9819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 -- 172.21.15.145:0/2358499655 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9819ff90 con 0x7efd98101a60 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9d247700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd8800d980 tx=0x7efd8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88017070 con 0x7efd98101a60 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd981a0280 con 0x7efd98101a60 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd88005020 con 0x7efd98101a60 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8800fbf0 con 0x7efd98101a60 2024-03-28T18:31:14.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.712+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd981a07c0 con 0x7efd98101a60 2024-03-28T18:31:14.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.713+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98068e50 con 0x7efd98101a60 2024-03-28T18:31:14.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.715+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7efd880051d0 con 0x7efd98101a60 2024-03-28T18:31:14.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.716+0000 7efd8effd700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 0x7efd840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:14.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.716+0000 7efd9da48700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 0x7efd840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:14.717 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.716+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7efd880cd770 con 0x7efd98101a60 2024-03-28T18:31:14.717 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.716+0000 7efd9da48700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 0x7efd840a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd981018c0 tx=0x7efd94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:31:14.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.719+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efd88020020 con 0x7efd98101a60 2024-03-28T18:31:14.942 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:14.943+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7efd980625d0 con 0x7efd840a2800 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi192 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:31:14.943 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:14 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:15.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi192 2024-03-28T18:31:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-28T18:31:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:31:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:14 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi192 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:31:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:14 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:31:15.276 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48585]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:15.276 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:15.276 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48585]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:15.276 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48585]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:15.680 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:15.680 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:15.680 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48588]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:15.680 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:31:15 smithi145 sudo[48588]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:15.870 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44701]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:15.870 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:15.870 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44701]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:15.870 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44701]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:15.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:15 smithi164 ceph-mon[36248]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:31:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:15 smithi145 ceph-mon[32151]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:16.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:16.135 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:16.135 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:15 smithi192 ceph-mon[36589]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:16.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:16.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:16.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44704]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:16.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:31:15 smithi164 sudo[44704]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:16.388 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:16.388 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:16.388 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:16.388 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45405]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:16.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:16.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:16.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:16.657 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:31:16 smithi192 sudo[45408]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:16.919 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48668]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:16.920 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48668]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:16.920 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48668]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:16.920 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48668]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:16.938 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:16.938+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efd88097c40 con 0x7efd98101a60 2024-03-28T18:31:17.175 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48692]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:17.175 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:17.175 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48692]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:17.175 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 sudo[48692]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:17.176 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 ceph-mon[32151]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:31:17.176 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.176 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.176 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:16 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:16 smithi192 ceph-mon[36589]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:16 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45411]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45411]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45411]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:17.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45411]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:16 smithi164 ceph-mon[36248]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:31:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:16 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:17.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:17.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:17.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:17.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 sudo[45414]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:17.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:31:17.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:17.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44707]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:17.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44707]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 sudo[44710]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]: dispatch 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]': finished 2024-03-28T18:31:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:31:18.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.273 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]: dispatch 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]': finished 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:31:18.274 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:18.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]: dispatch 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/466934903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "752a99f2-3b2f-4964-8e7c-1fd521d16fff"}]': finished 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T18:31:18.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:18 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/837160280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:18 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/837160280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:18 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/837160280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:20.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:19 smithi164 ceph-mon[36248]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:20.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:19 smithi145 ceph-mon[32151]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:20.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:19 smithi192 ceph-mon[36589]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:22.748 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:22 smithi145 ceph-mon[32151]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T18:31:22.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:22 smithi192 ceph-mon[36589]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T18:31:22.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:22 smithi164 ceph-mon[36248]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 6.0 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T18:31:24.703 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:24 smithi145 ceph-mon[32151]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T18:31:24.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:24 smithi192 ceph-mon[36589]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T18:31:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:24 smithi164 ceph-mon[36248]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T18:31:25.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:31:25.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:31:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:31:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:26 smithi145 ceph-mon[32151]: Deploying daemon osd.3 on smithi145 2024-03-28T18:31:26.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:26 smithi145 ceph-mon[32151]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-28T18:31:26.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:26 smithi192 ceph-mon[36589]: Deploying daemon osd.3 on smithi145 2024-03-28T18:31:26.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:26 smithi192 ceph-mon[36589]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-28T18:31:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:26 smithi164 ceph-mon[36248]: Deploying daemon osd.3 on smithi145 2024-03-28T18:31:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:26 smithi164 ceph-mon[36248]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-03-28T18:31:28.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:28 smithi145 ceph-mon[32151]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T18:31:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:28 smithi192 ceph-mon[36589]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T18:31:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:28 smithi164 ceph-mon[36248]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T18:31:30.739 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:30 smithi145 ceph-mon[32151]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-28T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:30 smithi192 ceph-mon[36589]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-28T18:31:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:30 smithi164 ceph-mon[36248]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 159 KiB/s wr, 10 op/s 2024-03-28T18:31:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:32 smithi192 ceph-mon[36589]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T18:31:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:32 smithi164 ceph-mon[36248]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T18:31:32.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:32.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:32.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:32 smithi145 ceph-mon[32151]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T18:31:34.771 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:34 smithi145 ceph-mon[32151]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:34.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.849+0000 7efd8effd700 1 -- 172.21.15.145:0/2358499655 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efd980625d0 con 0x7efd840a2800 2024-03-28T18:31:34.851 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 3 on host 'smithi145' 2024-03-28T18:31:34.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 msgr2=0x7efd840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:34.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 0x7efd840a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd981018c0 tx=0x7efd94009260 comp rx=0 tx=0).stop 2024-03-28T18:31:34.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 msgr2=0x7efd9819d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:34.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd8800d980 tx=0x7efd8800dc90 comp rx=0 tx=0).stop 2024-03-28T18:31:34.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 shutdown_connections 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd840a2800 0x7efd840a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd98102400 0x7efd9819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd98100860 0x7efd981037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 --2- 172.21.15.145:0/2358499655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd98101a60 0x7efd9819d080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 >> 172.21.15.145:0/2358499655 conn(0x7efd980fbde0 msgr2=0x7efd980fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 shutdown_connections 2024-03-28T18:31:34.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:31:34.852+0000 7efd9fcac700 1 -- 172.21.15.145:0/2358499655 wait complete. 2024-03-28T18:31:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:34 smithi192 ceph-mon[36589]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:34 smithi164 ceph-mon[36248]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:35.535 DEBUG:teuthology.orchestra.run.smithi145:osd.3> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T18:31:35.537 INFO:tasks.cephadm:Deploying osd.4 on smithi164 with /dev/vg_nvme/lv_3... 2024-03-28T18:31:35.537 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:31:35.784 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:31:35.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:35.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:35.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:35.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:35.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:31:35.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:35 smithi164 ceph-mon[36248]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:36.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:31:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:35 smithi192 ceph-mon[36589]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:31:36.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:35 smithi145 ceph-mon[32151]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:36.993 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-mon[32151]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:31:36.993 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-mon[32151]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:31:36.993 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-mon[32151]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:31:36.993 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:36.994 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:31:36.888+0000 7efda6952700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T18:31:36.994 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:31:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:31:36.894+0000 7efda2745700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:31:37.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:36 smithi164 ceph-mon[36248]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:31:37.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:36 smithi164 ceph-mon[36248]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:31:37.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:36 smithi164 ceph-mon[36248]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:31:37.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:36 smithi192 ceph-mon[36589]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:31:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:36 smithi192 ceph-mon[36589]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:31:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:36 smithi192 ceph-mon[36589]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:31:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114235 s, 918 MB/s 2024-03-28T18:31:37.568 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] boot 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:31:38.114 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.115 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:37 smithi145 ceph-mon[32151]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:38.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] boot 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:37 smithi192 ceph-mon[36589]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: from='osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: osd.3 [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] boot 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:31:38.163 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:37 smithi164 ceph-mon[36248]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:31:38.193 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_3 2024-03-28T18:31:38.436 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi145 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi145 to 8544M 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:31:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:31:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:31:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi145 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi145 to 8544M 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi145 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi145 to 8544M 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2888319160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f41037f0 msgr2=0x7fb3f4103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2888319160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f41037f0 0x7fb3f4103c10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009990 tx=0x7fb3e4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2888319160 shutdown_connections 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2888319160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f4100c10 0x7fb3f410a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2888319160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f41037f0 0x7fb3f4103c10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2888319160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 0x7fb3f41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2888319160 >> 172.21.15.164:0/2888319160 conn(0x7fb3f40fbe30 msgr2=0x7fb3f40fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2888319160 shutdown_connections 2024-03-28T18:31:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.100+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2888319160 wait complete. 2024-03-28T18:31:40.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:39 smithi164 ceph-mon[36248]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:31:40.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:39 smithi164 ceph-mon[36248]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 Processor -- start 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 -- start start 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 0x7fb3f4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f41037f0 0x7fb3f419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f410de40 con 0x7fb3f41037f0 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f410dcc0 con 0x7fb3f4100c10 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f8deb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f410dfc0 con 0x7fb3f4100270 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.101+0000 7fb3f1d9b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42556/0 (socket says 172.21.15.164:42556) 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 -- 172.21.15.164:0/2490668497 learned_addr learned my addr 172.21.15.164:0/2490668497 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f2d9d700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f41037f0 0x7fb3f419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f259c700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 0x7fb3f4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 -- 172.21.15.164:0/2490668497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 msgr2=0x7fb3f4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 0x7fb3f4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 -- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f41037f0 msgr2=0x7fb3f419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f41037f0 0x7fb3f419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 -- 172.21.15.164:0/2490668497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f41a0090 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f1d9b700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d980 tx=0x7fb3e800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8017070 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f41a0380 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e8005020 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e800fbf0 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.102+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f41a08c0 con 0x7fb3f4100c10 2024-03-28T18:31:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.103+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4068e50 con 0x7fb3f4100c10 2024-03-28T18:31:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.105+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb3e80051d0 con 0x7fb3f4100c10 2024-03-28T18:31:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.105+0000 7fb3e37fe700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 0x7fb3dc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:31:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.105+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3669+0+0 (secure 0 0 0) 0x7fb3e80cdaa0 con 0x7fb3f4100c10 2024-03-28T18:31:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.105+0000 7fb3f259c700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 0x7fb3dc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:31:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.106+0000 7fb3f259c700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 0x7fb3dc0a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000c00 tx=0x7fb3e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:31:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.109+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3e8020020 con 0x7fb3f4100c10 2024-03-28T18:31:40.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:39 smithi145 ceph-mon[32151]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:31:40.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:39 smithi145 ceph-mon[32151]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:40.314+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3f40625d0 con 0x7fb3dc0a2890 2024-03-28T18:31:40.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:39 smithi192 ceph-mon[36589]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:31:40.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:39 smithi192 ceph-mon[36589]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:41.193 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:40 smithi164 ceph-mon[36248]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:31:41.193 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:40 smithi164 ceph-mon[36248]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:41.193 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:41.193 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:41.193 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:40 smithi145 ceph-mon[32151]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:31:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:40 smithi145 ceph-mon[32151]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:41.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:41.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:40 smithi192 ceph-mon[36589]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:31:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:40 smithi192 ceph-mon[36589]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:31:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:31:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:31:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:42.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:41 smithi164 ceph-mon[36248]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:31:42.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:41 smithi164 ceph-mon[36248]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:31:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:41 smithi145 ceph-mon[32151]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:31:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:41 smithi145 ceph-mon[32151]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:31:42.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:41 smithi192 ceph-mon[36589]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:31:42.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:41 smithi192 ceph-mon[36589]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:31:43.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:42 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/1089453533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:42 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:42 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]': finished 2024-03-28T18:31:43.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:42 smithi164 ceph-mon[36248]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:31:43.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:31:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:42 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/1089453533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:42 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:42 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]': finished 2024-03-28T18:31:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:42 smithi145 ceph-mon[32151]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:31:43.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:31:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:42 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/1089453533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:42 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]: dispatch 2024-03-28T18:31:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:42 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fea7e9f1-4577-439a-81a8-5b6bf2e810e6"}]': finished 2024-03-28T18:31:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:42 smithi192 ceph-mon[36589]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T18:31:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:31:44.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:43 smithi145 ceph-mon[32151]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-28T18:31:44.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:43 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/2277979804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:44.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:43 smithi192 ceph-mon[36589]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-28T18:31:44.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:43 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/2277979804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:44.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:43 smithi164 ceph-mon[36248]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-28T18:31:44.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:43 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/2277979804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:31:46.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:46 smithi145 ceph-mon[32151]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:31:46.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:46 smithi164 ceph-mon[36248]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:31:46.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:46 smithi192 ceph-mon[36589]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:31:48.762 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:48 smithi164 ceph-mon[36248]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T18:31:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:48 smithi145 ceph-mon[32151]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T18:31:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:48 smithi192 ceph-mon[36589]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-28T18:31:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:50 smithi145 ceph-mon[32151]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:31:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:50.790 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:50 smithi164 ceph-mon[36248]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:50.790 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:31:50.790 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:50 smithi192 ceph-mon[36589]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:31:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:51.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:51 smithi145 ceph-mon[32151]: Deploying daemon osd.4 on smithi164 2024-03-28T18:31:51.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:51 smithi164 ceph-mon[36248]: Deploying daemon osd.4 on smithi164 2024-03-28T18:31:51.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:51 smithi192 ceph-mon[36589]: Deploying daemon osd.4 on smithi164 2024-03-28T18:31:52.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:52 smithi145 ceph-mon[32151]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:52.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:52 smithi192 ceph-mon[36589]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:52.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:52 smithi164 ceph-mon[36248]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:54 smithi145 ceph-mon[32151]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:54 smithi192 ceph-mon[36589]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:54.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:54 smithi164 ceph-mon[36248]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:56 smithi145 ceph-mon[32151]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:56.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:56 smithi164 ceph-mon[36248]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:56.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:56.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:56 smithi192 ceph-mon[36589]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:31:58.584 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:58.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:58.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:58.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:31:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:58.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:31:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:31:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:31:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:31:59.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.680+0000 7fb3e37fe700 1 -- 172.21.15.164:0/2490668497 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb3f40625d0 con 0x7fb3dc0a2890 2024-03-28T18:31:59.681 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 4 on host 'smithi164' 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.682+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 msgr2=0x7fb3dc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.682+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 0x7fb3dc0a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000c00 tx=0x7fb3e4005fb0 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.682+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 msgr2=0x7fb3f4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.682+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800d980 tx=0x7fb3e800dc90 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 shutdown_connections 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb3dc0a2890 0x7fb3dc0a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3f41037f0 0x7fb3f419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f4100270 0x7fb3f4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 --2- 172.21.15.164:0/2490668497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3f4100c10 0x7fb3f4198970 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 >> 172.21.15.164:0/2490668497 conn(0x7fb3f40fbe30 msgr2=0x7fb3f4106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 shutdown_connections 2024-03-28T18:31:59.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:31:59.683+0000 7fb3f8deb700 1 -- 172.21.15.164:0/2490668497 wait complete. 2024-03-28T18:32:00.347 DEBUG:teuthology.orchestra.run.smithi164:osd.4> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T18:32:00.350 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-03-28T18:32:00.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:32:00.594 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:00.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:00 smithi164 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:00 smithi192 ceph-mon[36589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:00.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:00.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:00.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:00 smithi145 ceph-mon[32151]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:32:01.794 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:32:01.794 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:01.795 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:32:01.795 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:01.795 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:01.795 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:01 smithi164 ceph-mon[36248]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:01.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:01 smithi192 ceph-mon[36589]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:02.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:32:02.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: from='osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:02.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:32:02.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:02.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:32:02.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:01 smithi145 ceph-mon[32151]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:02.462 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:32:02.141+0000 7f40e3d9f700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T18:32:02.462 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:32:02.148+0000 7f40dbb8a700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111322 s, 942 MB/s 2024-03-28T18:32:02.564 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-28T18:32:02.927 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:02 smithi145 ceph-mon[32151]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:32:02.927 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:02 smithi145 ceph-mon[32151]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:32:02.927 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:02.928 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:02 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:02.928 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:02 smithi145 ceph-mon[32151]: from='osd.4 ' entity='osd.4' 2024-03-28T18:32:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:32:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-mon[36248]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:32:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:02 smithi164 ceph-mon[36248]: from='osd.4 ' entity='osd.4' 2024-03-28T18:32:03.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:02 smithi192 ceph-mon[36589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:32:03.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:02 smithi192 ceph-mon[36589]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:32:03.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:02 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:02 smithi192 ceph-mon[36589]: from='osd.4 ' entity='osd.4' 2024-03-28T18:32:03.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-03-28T18:32:03.438 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:32:03.864 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] boot 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi164 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi164 to 8540M 2024-03-28T18:32:03.865 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.866 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:03.866 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:03.866 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:32:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:32:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] boot 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi164 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi164 to 8540M 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:03.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: osd.4 [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] boot 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi164 2024-03-28T18:32:04.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi164 to 8540M 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:04.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.000+0000 7fd2cf9da700 1 -- 172.21.15.192:0/3248866515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8102330 msgr2=0x7fd2c8102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.000+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/3248866515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8102330 0x7fd2c8102790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009990 tx=0x7fd2b8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.000+0000 7fd2cf9da700 1 -- 172.21.15.192:0/3248866515 shutdown_connections 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.000+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/3248866515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102cd0 0x7fd2c8109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.000+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/3248866515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8101130 0x7fd2c8101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/3248866515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8102330 0x7fd2c8102790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- 172.21.15.192:0/3248866515 >> 172.21.15.192:0/3248866515 conn(0x7fd2c80fc6f0 msgr2=0x7fd2c80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:05.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- 172.21.15.192:0/3248866515 shutdown_connections 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- 172.21.15.192:0/3248866515 wait complete. 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 Processor -- start 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- start start 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102330 0x7fd2c8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8102cd0 0x7fd2c819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810cc80 con 0x7fd2c8102330 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810cb00 con 0x7fd2c8102cd0 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cf9da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c810ce00 con 0x7fd2c8101130 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cd776700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cd776700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:33848/0 (socket says 172.21.15.192:33848) 2024-03-28T18:32:05.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.001+0000 7fd2cd776700 1 -- 172.21.15.192:0/985073334 learned_addr learned my addr 172.21.15.192:0/985073334 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2ccf75700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102330 0x7fd2c8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cdf77700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8102cd0 0x7fd2c819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 -- 172.21.15.192:0/985073334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8102cd0 msgr2=0x7fd2c819f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8102cd0 0x7fd2c819f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 -- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102330 msgr2=0x7fd2c8198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102330 0x7fd2c8198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 -- 172.21.15.192:0/985073334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c819ff90 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cd776700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400c9a0 tx=0x7fd2c400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4004220 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2c4007d40 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c81a0280 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4003da0 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.002+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c81a07c0 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.003+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c8068e50 con 0x7fd2c8101130 2024-03-28T18:32:05.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.004+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd2c4007860 con 0x7fd2c8101130 2024-03-28T18:32:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.005+0000 7fd2be7fc700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 0x7fd2b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.005+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fd2c40cd630 con 0x7fd2c8101130 2024-03-28T18:32:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.005+0000 7fd2ccf75700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 0x7fd2b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.006+0000 7fd2ccf75700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 0x7fd2b40a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800b5c0 tx=0x7fd2b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.010+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2c4014030 con 0x7fd2c8101130 2024-03-28T18:32:05.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:05.216+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2c80fe840 con 0x7fd2b40a2890 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:05 smithi164 ceph-mon[36248]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:06.007 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:05 smithi192 ceph-mon[36589]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:06.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:05 smithi145 ceph-mon[32151]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/760686876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]': finished 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:32:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:08 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/760686876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]': finished 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:32:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/760686876' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]: dispatch 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f883f88f-77ee-4a4e-9d4e-b7f0469c50fc"}]': finished 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T18:32:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:08 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:09.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/3631903387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:09.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:09 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/3631903387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:09.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:09 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/3631903387' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:10 smithi145 ceph-mon[32151]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:10.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:10 smithi192 ceph-mon[36589]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:10 smithi164 ceph-mon[36248]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:12.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:12 smithi192 ceph-mon[36589]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:12.764 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:12 smithi164 ceph-mon[36248]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:12 smithi145 ceph-mon[32151]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:14.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:14 smithi145 ceph-mon[32151]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:14.898 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:14 smithi192 ceph-mon[36589]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:14 smithi164 ceph-mon[36248]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:15.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:32:15.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:15.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:32:15.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:32:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:16.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:16 smithi145 ceph-mon[32151]: Deploying daemon osd.5 on smithi192 2024-03-28T18:32:16.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:16 smithi145 ceph-mon[32151]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:16 smithi192 ceph-mon[36589]: Deploying daemon osd.5 on smithi192 2024-03-28T18:32:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:16 smithi192 ceph-mon[36589]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:16 smithi164 ceph-mon[36248]: Deploying daemon osd.5 on smithi192 2024-03-28T18:32:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:16 smithi164 ceph-mon[36248]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:18.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:18 smithi145 ceph-mon[32151]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:18 smithi192 ceph-mon[36589]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:18 smithi164 ceph-mon[36248]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:20.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:20 smithi145 ceph-mon[32151]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:20 smithi192 ceph-mon[36589]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:20.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:20 smithi164 ceph-mon[36248]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:21 smithi164 ceph-mon[36248]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:21.979 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:21.980 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:21.980 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:21.980 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:21 smithi192 ceph-mon[36589]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:21 smithi145 ceph-mon[32151]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:23.739 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.734+0000 7fd2be7fc700 1 -- 172.21.15.192:0/985073334 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd2c80fe840 con 0x7fd2b40a2890 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 msgr2=0x7fd2b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 0x7fd2b40a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800b5c0 tx=0x7fd2b8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 msgr2=0x7fd2c8198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400c9a0 tx=0x7fd2c400ccb0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 shutdown_connections 2024-03-28T18:32:23.740 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd2b40a2890 0x7fd2b40a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2c8102cd0 0x7fd2c819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2c8101130 0x7fd2c8198330 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 --2- 172.21.15.192:0/985073334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2c8102330 0x7fd2c8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 >> 172.21.15.192:0/985073334 conn(0x7fd2c80fc6f0 msgr2=0x7fd2c80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 shutdown_connections 2024-03-28T18:32:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:32:23.736+0000 7fd2cf9da700 1 -- 172.21.15.192:0/985073334 wait complete. 2024-03-28T18:32:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:24 smithi192 ceph-mon[36589]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:24.612 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T18:32:24.614 INFO:tasks.cephadm:Deploying osd.6 on smithi145 with /dev/vg_nvme/lv_2... 2024-03-28T18:32:24.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:32:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:24 smithi145 ceph-mon[32151]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:24.864 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:32:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:24 smithi164 ceph-mon[36248]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:25 smithi145 ceph-mon[32151]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:25 smithi145 ceph-mon[32151]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:25 smithi192 ceph-mon[36589]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:25 smithi192 ceph-mon[36589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:25 smithi164 ceph-mon[36248]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:25 smithi164 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:32:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:26.630 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:26.630 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:32:26.630 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:32:26.630 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:26.630 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:26.631 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-mon[36589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:26.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:26 smithi145 ceph-mon[32151]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:26.840 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:32:26.840 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:32:26.840 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-28T18:32:26.840 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-28T18:32:26.840 INFO:teuthology.orchestra.run.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147697 s, 710 MB/s 2024-03-28T18:32:26.841 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: from='osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:26 smithi164 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:32:27.407 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:32:26.972+0000 7f68dffd6700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T18:32:27.407 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:32:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:32:26.980+0000 7f68d7dc1700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:32:27.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_2 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:27 smithi192 ceph-mon[36589]: from='osd.5 ' entity='osd.5' 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:27 smithi145 ceph-mon[32151]: from='osd.5 ' entity='osd.5' 2024-03-28T18:32:27.838 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:27 smithi164 ceph-mon[36248]: from='osd.5 ' entity='osd.5' 2024-03-28T18:32:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:32:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] boot 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:28.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] boot 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: osd.5 [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] boot 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:28.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:29.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.616+0000 7f5e3e041700 1 -- 172.21.15.145:0/1806811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 msgr2=0x7f5e38103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- 172.21.15.145:0/1806811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38103820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e200099c0 tx=0x7f5e20009cd0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- 172.21.15.145:0/1806811 shutdown_connections 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- 172.21.15.145:0/1806811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e38103d60 0x7f5e3810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- 172.21.15.145:0/1806811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e381021c0 0x7f5e381025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- 172.21.15.145:0/1806811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38103820 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- 172.21.15.145:0/1806811 >> 172.21.15.145:0/1806811 conn(0x7f5e380fd7a0 msgr2=0x7f5e380ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- 172.21.15.145:0/1806811 shutdown_connections 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- 172.21.15.145:0/1806811 wait complete. 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 Processor -- start 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- start start 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 0x7f5e38075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:29.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 0x7f5e38074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3810dd80 con 0x7f5e381033c0 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3810dc00 con 0x7f5e381021c0 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.617+0000 7f5e3e041700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3810df00 con 0x7f5e38103d60 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56836/0 (socket says 172.21.15.145:56836) 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 -- 172.21.15.145:0/2689068744 learned_addr learned my addr 172.21.15.145:0/2689068744 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e37fff700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 0x7f5e38074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e377fe700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 0x7f5e38075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 -- 172.21.15.145:0/2689068744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 msgr2=0x7f5e38074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 0x7f5e38074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 -- 172.21.15.145:0/2689068744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 msgr2=0x7f5e38075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:29.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 0x7f5e38075f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 -- 172.21.15.145:0/2689068744 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38075370 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e37fff700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 0x7f5e38074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e377fe700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 0x7f5e38075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e2ffff700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e20003890 tx=0x7f5e20003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20027070 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e381a2500 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e20022470 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000f6c0 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.618+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e381a2950 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.619+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e38068e50 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.620+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5e2002a070 con 0x7f5e381033c0 2024-03-28T18:32:29.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.621+0000 7f5e357fa700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 0x7f5e240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:29.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.621+0000 7f5e377fe700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 0x7f5e240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:29.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.621+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f5e200cec90 con 0x7f5e381033c0 2024-03-28T18:32:29.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.621+0000 7f5e377fe700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 0x7f5e240a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e28005fd0 tx=0x7f5e28005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:29.625 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.626+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e2005e740 con 0x7f5e381033c0 2024-03-28T18:32:29.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:29 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi192 2024-03-28T18:32:29.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:29 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-28T18:32:29.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:29 smithi145 ceph-mon[32151]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:32:29.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:29.831+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e380625d0 con 0x7f5e240a2890 2024-03-28T18:32:29.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:29 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi192 2024-03-28T18:32:29.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:29 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-28T18:32:29.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:29 smithi192 ceph-mon[36589]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:32:29.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:29 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi192 2024-03-28T18:32:29.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:29 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-28T18:32:29.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:29 smithi164 ceph-mon[36248]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:32:30.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:30 smithi145 ceph-mon[32151]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:30.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:30.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:30.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:30.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:30 smithi145 ceph-mon[32151]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:32:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:30 smithi192 ceph-mon[36589]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:30 smithi192 ceph-mon[36589]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:32:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:30 smithi164 ceph-mon[36248]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:30 smithi164 ceph-mon[36248]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:32:31.772 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:31 smithi145 ceph-mon[32151]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:31.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:31 smithi192 ceph-mon[36589]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:31 smithi164 ceph-mon[36248]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:32.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:32 smithi145 ceph-mon[32151]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:32.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:32 smithi145 ceph-mon[32151]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T18:32:32.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:32 smithi145 ceph-mon[32151]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:32:32.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:32 smithi192 ceph-mon[36589]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:32.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:32 smithi192 ceph-mon[36589]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T18:32:32.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:32 smithi192 ceph-mon[36589]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:32:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:32 smithi164 ceph-mon[36248]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:32 smithi164 ceph-mon[36248]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T18:32:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:32 smithi164 ceph-mon[36248]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:32:33.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:32:33.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]: dispatch 2024-03-28T18:32:33.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]': finished 2024-03-28T18:32:33.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:32:33.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:33.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:33 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2873767901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:33.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:32:33.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]: dispatch 2024-03-28T18:32:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]': finished 2024-03-28T18:32:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:32:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:33 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2873767901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]: dispatch 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3980928167' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf2bf55a-48f0-46b4-acbd-f8c012e9b39d"}]': finished 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:33 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2873767901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:34 smithi192 ceph-mon[36589]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:34 smithi164 ceph-mon[36248]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:34 smithi145 ceph-mon[32151]: pgmap v144: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T18:32:35.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:35 smithi164 ceph-mon[36248]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:32:35.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:35 smithi164 ceph-mon[36248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T18:32:35.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:35 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:32:36.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:35 smithi145 ceph-mon[32151]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:32:36.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:35 smithi145 ceph-mon[32151]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T18:32:36.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:35 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:32:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:35 smithi192 ceph-mon[36589]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:32:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:35 smithi192 ceph-mon[36589]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T18:32:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:35 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:32:38.749 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:38 smithi145 ceph-mon[32151]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:32:38.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:38 smithi192 ceph-mon[36589]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:32:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:38 smithi164 ceph-mon[36248]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:32:40.743 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:40 smithi145 ceph-mon[32151]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-28T18:32:40.743 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:32:40.744 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:40 smithi192 ceph-mon[36589]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-28T18:32:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:32:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:40 smithi164 ceph-mon[36248]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-28T18:32:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:32:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:41 smithi145 ceph-mon[32151]: Deploying daemon osd.6 on smithi145 2024-03-28T18:32:41.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:41 smithi192 ceph-mon[36589]: Deploying daemon osd.6 on smithi145 2024-03-28T18:32:41.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:41 smithi164 ceph-mon[36248]: Deploying daemon osd.6 on smithi145 2024-03-28T18:32:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:42 smithi145 ceph-mon[32151]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-28T18:32:42.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:42 smithi164 ceph-mon[36248]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-28T18:32:42.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:42 smithi192 ceph-mon[36589]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-28T18:32:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:44 smithi145 ceph-mon[32151]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-28T18:32:44.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:44 smithi192 ceph-mon[36589]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-28T18:32:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:44 smithi164 ceph-mon[36248]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-28T18:32:46.748 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:46 smithi145 ceph-mon[32151]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T18:32:46.748 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:46.748 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:46.748 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:46 smithi192 ceph-mon[36589]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T18:32:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:46 smithi164 ceph-mon[36248]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-28T18:32:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:48.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:48 smithi145 ceph-mon[32151]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:48 smithi192 ceph-mon[36589]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:48 smithi164 ceph-mon[36248]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:48.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.996+0000 7f5e357fa700 1 -- 172.21.15.145:0/2689068744 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5e380625d0 con 0x7f5e240a2890 2024-03-28T18:32:48.996 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 6 on host 'smithi145' 2024-03-28T18:32:48.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 msgr2=0x7f5e240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:48.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 0x7f5e240a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e28005fd0 tx=0x7f5e28005dc0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 msgr2=0x7f5e38074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e20003890 tx=0x7f5e20003730 comp rx=0 tx=0).stop 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 shutdown_connections 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5e240a2890 0x7f5e240a4d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e38103d60 0x7f5e38074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e381021c0 0x7f5e38075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 --2- 172.21.15.145:0/2689068744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e381033c0 0x7f5e38074580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 >> 172.21.15.145:0/2689068744 conn(0x7f5e380fd7a0 msgr2=0x7f5e38106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 shutdown_connections 2024-03-28T18:32:48.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:32:48.998+0000 7f5e3e041700 1 -- 172.21.15.145:0/2689068744 wait complete. 2024-03-28T18:32:49.959 DEBUG:teuthology.orchestra.run.smithi145:osd.6> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T18:32:49.962 INFO:tasks.cephadm:Deploying osd.7 on smithi164 with /dev/vg_nvme/lv_2... 2024-03-28T18:32:49.962 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:32:50.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:49 smithi145 ceph-mon[32151]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:50.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:49 smithi145 ceph-mon[32151]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:32:50.211 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:32:50.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:49 smithi164 ceph-mon[36248]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:50.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:49 smithi164 ceph-mon[36248]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:32:50.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:50.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:49 smithi192 ceph-mon[36589]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:50.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:49 smithi192 ceph-mon[36589]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:51.137 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:51.138 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:32:51.006+0000 7f0cd8111700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T18:32:51.138 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:32:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:32:51.014+0000 7f0cd3f04700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:51.360 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:52.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:52 smithi145 ceph-mon[32151]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:32:52.109 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:52 smithi145 ceph-mon[32151]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:32:52.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.110 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:52 smithi145 ceph-mon[32151]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129121 s, 812 MB/s 2024-03-28T18:32:52.347 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-28T18:32:52.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:52 smithi164 ceph-mon[36248]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:32:52.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:52 smithi164 ceph-mon[36248]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:32:52.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:52 smithi164 ceph-mon[36248]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:52.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:52 smithi192 ceph-mon[36589]: from='osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:32:52.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:52 smithi192 ceph-mon[36589]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:32:52.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:52.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:52 smithi192 ceph-mon[36589]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:32:53.007 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_2 2024-03-28T18:32:53.259 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] boot 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:32:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:53.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:32:53.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:32:53.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:53.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] boot 2024-03-28T18:32:53.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:32:53.385 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: osd.6 [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] boot 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:32:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:32:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:32:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:54.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:32:54.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:54.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:54.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:54.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:55.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 -- 172.21.15.164:0/3659874730 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 msgr2=0x7f0554101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:55.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 --2- 172.21.15.164:0/3659874730 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f0554101510 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f054c00b0d0 tx=0x7f054c00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 -- 172.21.15.164:0/3659874730 shutdown_connections 2024-03-28T18:32:55.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 --2- 172.21.15.164:0/3659874730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0554102c90 0x7f0554109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 --2- 172.21.15.164:0/3659874730 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f0554101510 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 --2- 172.21.15.164:0/3659874730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05541022f0 0x7f0554102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 -- 172.21.15.164:0/3659874730 >> 172.21.15.164:0/3659874730 conn(0x7f05540fc690 msgr2=0x7f05540fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 -- 172.21.15.164:0/3659874730 shutdown_connections 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.033+0000 7f055bb34700 1 -- 172.21.15.164:0/3659874730 wait complete. 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 Processor -- start 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 -- start start 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05541022f0 0x7f0554198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 0x7f055419f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410cce0 con 0x7f0554102c90 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410cb60 con 0x7f05541010f0 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410ce60 con 0x7f05541022f0 2024-03-28T18:32:55.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42374/0 (socket says 172.21.15.164:42374) 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 -- 172.21.15.164:0/3195125772 learned_addr learned my addr 172.21.15.164:0/3195125772 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055a0d1700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 0x7f055419f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05590cf700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05541022f0 0x7f0554198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 -- 172.21.15.164:0/3195125772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05541022f0 msgr2=0x7f0554198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05541022f0 0x7f0554198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 -- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 msgr2=0x7f055419f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 0x7f055419f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 -- 172.21.15.164:0/3195125772 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05541a0040 con 0x7f05541010f0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055a0d1700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 0x7f055419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f05598d0700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f054c000f80 tx=0x7f054c007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c021020 con 0x7f05541010f0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05541a02d0 con 0x7f05541010f0 2024-03-28T18:32:55.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f054c01d950 con 0x7f05541010f0 2024-03-28T18:32:55.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.034+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c005510 con 0x7f05541010f0 2024-03-28T18:32:55.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.035+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05541a0810 con 0x7f05541010f0 2024-03-28T18:32:55.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.036+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0554068e50 con 0x7f05541010f0 2024-03-28T18:32:55.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.037+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f054c024020 con 0x7f05541010f0 2024-03-28T18:32:55.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.037+0000 7f0552ffd700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 0x7f053c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:55.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.037+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f054c00f020 con 0x7f05541010f0 2024-03-28T18:32:55.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.037+0000 7f05590cf700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 0x7f053c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:55.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.038+0000 7f05590cf700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 0x7f053c0a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0548005950 tx=0x7f05480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:55.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.042+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f054c09cea0 con 0x7f05541010f0 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi145 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi145 to 5696M 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:55 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:55.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:32:55.249+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f055406af40 con 0x7f053c0a2890 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi145 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi145 to 5696M 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:55 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:55.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi145 2024-03-28T18:32:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi145 to 5696M 2024-03-28T18:32:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:32:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:55 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:32:56.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:56 smithi164 ceph-mon[36248]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:56.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:56.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:56.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:56.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:56 smithi164 ceph-mon[36248]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:56.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:56 smithi192 ceph-mon[36589]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:56.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:56.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:56.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:56.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:56 smithi192 ceph-mon[36589]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:56.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:56 smithi145 ceph-mon[32151]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:56.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:32:56.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:32:56.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:56.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:56 smithi145 ceph-mon[32151]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/675616069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]': finished 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:32:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:32:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/675616069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]': finished 2024-03-28T18:32:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:32:58.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/675616069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]: dispatch 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "532710ac-62e7-479f-885d-5604af92fd37"}]': finished 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:32:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:32:59.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:32:59 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/1367141797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:59.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:32:59 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/1367141797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:32:59.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:32:59 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/1367141797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:00 smithi145 ceph-mon[32151]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:00.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:00 smithi192 ceph-mon[36589]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:00 smithi164 ceph-mon[36248]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:02.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:02 smithi164 ceph-mon[36248]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:02.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:02 smithi145 ceph-mon[32151]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:02.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:02 smithi192 ceph-mon[36589]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:04 smithi164 ceph-mon[36248]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:04 smithi145 ceph-mon[32151]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:04 smithi192 ceph-mon[36589]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:05.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:33:05.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:05 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:05.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:33:05.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:05 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:05.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:33:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:05 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:06.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:06 smithi164 ceph-mon[36248]: Deploying daemon osd.7 on smithi164 2024-03-28T18:33:06.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:06 smithi164 ceph-mon[36248]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:06 smithi145 ceph-mon[32151]: Deploying daemon osd.7 on smithi164 2024-03-28T18:33:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:06 smithi145 ceph-mon[32151]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:06.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:06 smithi192 ceph-mon[36589]: Deploying daemon osd.7 on smithi164 2024-03-28T18:33:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:06 smithi192 ceph-mon[36589]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:08.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:08 smithi164 ceph-mon[36248]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:08 smithi145 ceph-mon[32151]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:08.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:08 smithi192 ceph-mon[36589]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:10.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:10 smithi164 ceph-mon[36248]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:10 smithi145 ceph-mon[32151]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:10.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:10 smithi192 ceph-mon[36589]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:12.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:12.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:12 smithi164 ceph-mon[36248]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:12.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:12.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:12 smithi145 ceph-mon[32151]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:12.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:12.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:12.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:12 smithi192 ceph-mon[36589]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:14.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:14 smithi145 ceph-mon[32151]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:14.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:14 smithi164 ceph-mon[36248]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:14.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:14 smithi192 ceph-mon[36589]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:14.920 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 7 on host 'smithi164' 2024-03-28T18:33:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.920+0000 7f0552ffd700 1 -- 172.21.15.164:0/3195125772 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f055406af40 con 0x7f053c0a2890 2024-03-28T18:33:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 msgr2=0x7f053c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 0x7f053c0a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0548005950 tx=0x7f05480058e0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 msgr2=0x7f05541983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f054c000f80 tx=0x7f054c007dc0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 shutdown_connections 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f053c0a2890 0x7f053c0a4d50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554102c90 0x7f055419f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05541010f0 0x7f05541983e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 --2- 172.21.15.164:0/3195125772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05541022f0 0x7f0554198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 >> 172.21.15.164:0/3195125772 conn(0x7f05540fc690 msgr2=0x7f055406a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 shutdown_connections 2024-03-28T18:33:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:33:14.922+0000 7f055bb34700 1 -- 172.21.15.164:0/3195125772 wait complete. 2024-03-28T18:33:15.632 DEBUG:teuthology.orchestra.run.smithi164:osd.7> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T18:33:15.635 INFO:tasks.cephadm:Deploying osd.8 on smithi192 with /dev/vg_nvme/lv_2... 2024-03-28T18:33:15.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:33:15.882 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:16.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:16 smithi164 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:16 smithi164 ceph-mon[36248]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:16 smithi164 ceph-mon[36248]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:16 smithi164 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:17.225 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:16 smithi192 ceph-mon[36589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:33:17.225 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:16 smithi192 ceph-mon[36589]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:33:17.225 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:17.225 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:16 smithi192 ceph-mon[36589]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:17.225 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:16 smithi192 ceph-mon[36589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:16 smithi145 ceph-mon[32151]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:33:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:16 smithi145 ceph-mon[32151]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:33:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:16 smithi145 ceph-mon[32151]: from='osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:16 smithi145 ceph-mon[32151]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:33:17.670 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:33:17.341+0000 7fc6925b0700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T18:33:17.670 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:33:17.352+0000 7fc68a39b700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:33:17.939 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:33:17.939 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:33:17.939 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-28T18:33:17.939 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-28T18:33:17.939 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114316 s, 917 MB/s 2024-03-28T18:33:17.940 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 2024-03-28T18:33:17.940 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-28T18:33:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:33:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:33:18.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T18:33:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: from='osd.7 ' entity='osd.7' 2024-03-28T18:33:18.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:17 smithi164 ceph-mon[36248]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:18.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: from='osd.7 ' entity='osd.7' 2024-03-28T18:33:18.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:17 smithi145 ceph-mon[32151]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:18.278 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:33:18.278 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:33:18.278 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: osdmap e52: 8 total, 7 up, 8 in 2024-03-28T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: from='osd.7 ' entity='osd.7' 2024-03-28T18:33:18.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:17 smithi192 ceph-mon[36589]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:33:18.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-03-28T18:33:18.807 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] boot 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:33:19.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] boot 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:33:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: osd.7 [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] boot 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:33:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:33:20.106 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:33:20.106 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi164 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:33:20.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:20.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi164 2024-03-28T18:33:20.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.331+0000 7f6d30093700 1 -- 172.21.15.192:0/3798541850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28107f40 msgr2=0x7f6d2810a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.331+0000 7f6d30093700 1 --2- 172.21.15.192:0/3798541850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28107f40 0x7f6d2810a350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000b0a0 tx=0x7f6d2000b3b0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- 172.21.15.192:0/3798541850 shutdown_connections 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- 172.21.15.192:0/3798541850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28107f40 0x7f6d2810a350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- 172.21.15.192:0/3798541850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 0x7f6d281050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- 172.21.15.192:0/3798541850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28105610 0x7f6d28107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- 172.21.15.192:0/3798541850 >> 172.21.15.192:0/3798541850 conn(0x7f6d280fc6b0 msgr2=0x7f6d280fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- 172.21.15.192:0/3798541850 shutdown_connections 2024-03-28T18:33:20.333 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- 172.21.15.192:0/3798541850 wait complete. 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 Processor -- start 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- start start 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 0x7f6d28198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28107f40 0x7f6d2819fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2810dea0 con 0x7f6d28107f40 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2810dd20 con 0x7f6d28102ce0 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.332+0000 7f6d30093700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2810e020 con 0x7f6d28105610 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:20.336 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:37292/0 (socket says 172.21.15.192:37292) 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 -- 172.21.15.192:0/753514274 learned_addr learned my addr 172.21.15.192:0/753514274 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2de2f700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 0x7f6d28198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2e630700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28107f40 0x7f6d2819fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 -- 172.21.15.192:0/753514274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 msgr2=0x7f6d28198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 0x7f6d28198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 -- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28107f40 msgr2=0x7f6d2819fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28107f40 0x7f6d2819fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 -- 172.21.15.192:0/753514274 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d281a00b0 con 0x7f6d28105610 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d2d62e700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b560 tx=0x7f6d1800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18015020 con 0x7f6d28105610 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d281a03a0 con 0x7f6d28105610 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d18011d10 con 0x7f6d28105610 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18010e50 con 0x7f6d28105610 2024-03-28T18:33:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.333+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d281a09b0 con 0x7f6d28105610 2024-03-28T18:33:20.338 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.334+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d28068e50 con 0x7f6d28105610 2024-03-28T18:33:20.338 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.335+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6d18018020 con 0x7f6d28105610 2024-03-28T18:33:20.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.336+0000 7f6d1effd700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 0x7f6d140a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:20.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.336+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f6d180cdb00 con 0x7f6d28105610 2024-03-28T18:33:20.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.336+0000 7f6d2de2f700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 0x7f6d140a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:20.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.337+0000 7f6d2de2f700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 0x7f6d140a4ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009a10 tx=0x7f6d24009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:20.340 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.340+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d1801f020 con 0x7f6d28105610 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi164 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:20.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:20.545 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:20.544+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d280625d0 con 0x7f6d140a2a30 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi164 to 5693M 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:21.264 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi164 to 5693M 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi164 to 5693M 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:22.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:21 smithi192 ceph-mon[36589]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:33:22.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:21 smithi192 ceph-mon[36589]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:22.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:21 smithi145 ceph-mon[32151]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:33:22.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:21 smithi145 ceph-mon[32151]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:21 smithi164 ceph-mon[36248]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:33:22.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:21 smithi164 ceph-mon[36248]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:23.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:22 smithi145 ceph-mon[32151]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T18:33:23.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:22 smithi192 ceph-mon[36589]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T18:33:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:22 smithi164 ceph-mon[36248]: osdmap e57: 8 total, 8 up, 8 in 2024-03-28T18:33:24.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/2495101947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]': finished 2024-03-28T18:33:24.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:33:24.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:24.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:24.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:23 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/346105512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/2495101947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]': finished 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:24.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:23 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/346105512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/2495101947' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]: dispatch 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3ae89b-0840-440c-8bce-854d680e9538"}]': finished 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:24.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:23 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/346105512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:26.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:26 smithi145 ceph-mon[32151]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:26.853 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:26 smithi192 ceph-mon[36589]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:26 smithi164 ceph-mon[36248]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:28 smithi145 ceph-mon[32151]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:28.845 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:28 smithi192 ceph-mon[36589]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:28 smithi164 ceph-mon[36248]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:30.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:30 smithi192 ceph-mon[36589]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:30 smithi145 ceph-mon[32151]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:30 smithi164 ceph-mon[36248]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:31.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:33:31.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:31.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:31 smithi145 ceph-mon[32151]: Deploying daemon osd.8 on smithi192 2024-03-28T18:33:31.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:31 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:33:31.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:31 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:31.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:31 smithi192 ceph-mon[36589]: Deploying daemon osd.8 on smithi192 2024-03-28T18:33:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:31 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:33:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:31 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:31 smithi164 ceph-mon[36248]: Deploying daemon osd.8 on smithi192 2024-03-28T18:33:32.672 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:32 smithi192 ceph-mon[36589]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:32.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:32 smithi145 ceph-mon[32151]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:32 smithi164 ceph-mon[36248]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:34.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:34 smithi192 ceph-mon[36589]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:34.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:34 smithi145 ceph-mon[32151]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:34 smithi164 ceph-mon[36248]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:36.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:36 smithi192 ceph-mon[36589]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:36.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.664 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:36 smithi145 ceph-mon[32151]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:36 smithi164 ceph-mon[36248]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:38.604 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:38 smithi192 ceph-mon[36589]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:38.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:38 smithi145 ceph-mon[32151]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:38 smithi164 ceph-mon[36248]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:39.459 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.458+0000 7f6d1effd700 1 -- 172.21.15.192:0/753514274 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6d280625d0 con 0x7f6d140a2a30 2024-03-28T18:33:39.459 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 8 on host 'smithi192' 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 msgr2=0x7f6d140a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 0x7f6d140a4ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009a10 tx=0x7f6d24009190 comp rx=0 tx=0).stop 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 msgr2=0x7f6d28198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800b560 tx=0x7f6d1800b870 comp rx=0 tx=0).stop 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 shutdown_connections 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6d140a2a30 0x7f6d140a4ef0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d28107f40 0x7f6d2819fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.461 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d28102ce0 0x7f6d28198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 --2- 172.21.15.192:0/753514274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d28105610 0x7f6d28198990 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 >> 172.21.15.192:0/753514274 conn(0x7f6d280fc6b0 msgr2=0x7f6d281035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:39.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 shutdown_connections 2024-03-28T18:33:39.462 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:33:39.461+0000 7f6d30093700 1 -- 172.21.15.192:0/753514274 wait complete. 2024-03-28T18:33:40.038 DEBUG:teuthology.orchestra.run.smithi192:osd.8> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T18:33:40.040 INFO:tasks.cephadm:Deploying osd.9 on smithi145 with /dev/vg_nvme/lv_1... 2024-03-28T18:33:40.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:33:40.291 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.659 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:40.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.772 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:40.772 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.772 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:33:40.772 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:40.772 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:41 smithi164 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:33:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:41 smithi164 ceph-mon[36248]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:33:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:41 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:41 smithi164 ceph-mon[36248]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:41 smithi164 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:41.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-mon[36589]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:33:41.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-mon[36589]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:33:41.759 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:41.759 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-mon[36589]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:41.759 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-mon[36589]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:41 smithi145 ceph-mon[32151]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:33:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:41 smithi145 ceph-mon[32151]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:33:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:41 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:41 smithi145 ceph-mon[32151]: from='osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:41 smithi145 ceph-mon[32151]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:33:42.078 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:33:41.880+0000 7ff616b35700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T18:33:42.078 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:33:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:33:41.890+0000 7ff60e920700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122475 s, 856 MB/s 2024-03-28T18:33:42.579 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: from='osd.8 ' entity='osd.8' 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] boot 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:33:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:33:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: from='osd.8 ' entity='osd.8' 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] boot 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:33:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: from='osd.8 ' entity='osd.8' 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: osd.8 [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] boot 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:33:42.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:33:43.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi145:vg_nvme/lv_1 2024-03-28T18:33:43.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:43 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:33:43.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:43 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:33:43.528 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:43 smithi192 ceph-mon[36589]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:33:43.640 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:43 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:33:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:43 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:33:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:43 smithi164 ceph-mon[36248]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:33:43.761 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:43 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:33:43.762 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:43 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:33:43.762 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:43 smithi145 ceph-mon[32151]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:33:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi192 2024-03-28T18:33:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:44 smithi145 ceph-mon[32151]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi192 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:44 smithi192 ceph-mon[36589]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi192 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:33:44.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:44 smithi164 ceph-mon[36248]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:33:45.468 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 -- 172.21.15.145:0/2670391488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e14102c60 msgr2=0x7f7e14109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:45.468 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 --2- 172.21.15.145:0/2670391488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e14102c60 0x7f7e14109510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00b0d0 tx=0x7f7e0c00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.468 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 -- 172.21.15.145:0/2670391488 shutdown_connections 2024-03-28T18:33:45.468 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 --2- 172.21.15.145:0/2670391488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e14102c60 0x7f7e14109510 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.468 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 --2- 172.21.15.145:0/2670391488 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141010c0 0x7f7e141014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 --2- 172.21.15.145:0/2670391488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 0x7f7e14102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 -- 172.21.15.145:0/2670391488 >> 172.21.15.145:0/2670391488 conn(0x7f7e140fc660 msgr2=0x7f7e140feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 -- 172.21.15.145:0/2670391488 shutdown_connections 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.465+0000 7f7e1a298700 1 -- 172.21.15.145:0/2670391488 wait complete. 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 Processor -- start 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 -- start start 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 0x7f7e1419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 0x7f7e141a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1410cd00 con 0x7f7e141010c0 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1410cb80 con 0x7f7e14102c60 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e1a298700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1410ce80 con 0x7f7e141022c0 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:45.469 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52676/0 (socket says 172.21.15.145:52676) 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 -- 172.21.15.145:0/3028592414 learned_addr learned my addr 172.21.15.145:0/3028592414 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e137fe700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 0x7f7e1419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e18835700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 0x7f7e141a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 -- 172.21.15.145:0/3028592414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 msgr2=0x7f7e1419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 0x7f7e1419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 -- 172.21.15.145:0/3028592414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 msgr2=0x7f7e141a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 0x7f7e141a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 -- 172.21.15.145:0/3028592414 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e141a2200 con 0x7f7e141010c0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e18835700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 0x7f7e141a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.466+0000 7f7e13fff700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c9c0 tx=0x7f7dfc00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.467+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc007ac0 con 0x7f7e141010c0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.467+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dfc007c60 con 0x7f7e141010c0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.467+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc018660 con 0x7f7e141010c0 2024-03-28T18:33:45.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.467+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e141a24f0 con 0x7f7e141010c0 2024-03-28T18:33:45.471 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.467+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e141a2a60 con 0x7f7e141010c0 2024-03-28T18:33:45.471 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.468+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e14068e50 con 0x7f7e141010c0 2024-03-28T18:33:45.471 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.469+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7dfc00f450 con 0x7f7e141010c0 2024-03-28T18:33:45.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.469+0000 7f7e117fa700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 0x7f7e000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:45.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.469+0000 7f7e137fe700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 0x7f7e000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:45.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.469+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f7dfc0ce0f0 con 0x7f7e141010c0 2024-03-28T18:33:45.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.470+0000 7f7e137fe700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 0x7f7e000a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009a10 tx=0x7f7e04009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:45.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.474+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7dfc014030 con 0x7f7e141010c0 2024-03-28T18:33:45.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:33:45.679+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e140fe760 con 0x7f7e000a2890 2024-03-28T18:33:46.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:46 smithi145 ceph-mon[32151]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:33:46.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:46 smithi145 ceph-mon[32151]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:46.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:46.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:46.540 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:46 smithi164 ceph-mon[36248]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:33:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:46 smithi164 ceph-mon[36248]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:46 smithi192 ceph-mon[36589]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:33:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:46 smithi192 ceph-mon[36589]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:33:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:33:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:47.698 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:47 smithi145 ceph-mon[32151]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:47.699 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:47 smithi145 ceph-mon[32151]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T18:33:47.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:47 smithi164 ceph-mon[36248]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:47.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:47 smithi164 ceph-mon[36248]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T18:33:47.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:47 smithi192 ceph-mon[36589]: from='client.14409 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:47.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:47 smithi192 ceph-mon[36589]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T18:33:48.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:48 smithi164 ceph-mon[36248]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:48 smithi145 ceph-mon[32151]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:48 smithi192 ceph-mon[36589]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:33:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:49 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]: dispatch 2024-03-28T18:33:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:49 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]': finished 2024-03-28T18:33:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:49 smithi164 ceph-mon[36248]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:33:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:33:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:49 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1654593891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:49 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]: dispatch 2024-03-28T18:33:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:49 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]': finished 2024-03-28T18:33:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:49 smithi145 ceph-mon[32151]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:33:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:33:49.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:49 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1654593891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:49.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:49 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]: dispatch 2024-03-28T18:33:49.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:49 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2989569696' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c299158-d952-43b7-8ef1-6da9ab558f86"}]': finished 2024-03-28T18:33:49.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:49 smithi192 ceph-mon[36589]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:33:49.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:33:49.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:49 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1654593891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:33:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:50 smithi164 ceph-mon[36248]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:33:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:50 smithi145 ceph-mon[32151]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:33:50.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:50 smithi192 ceph-mon[36589]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-28T18:33:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:52 smithi164 ceph-mon[36248]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:33:52.713 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:52 smithi145 ceph-mon[32151]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:33:52.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:52 smithi192 ceph-mon[36589]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:33:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:54 smithi164 ceph-mon[36248]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:33:54.738 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:54 smithi145 ceph-mon[32151]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:33:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:54 smithi192 ceph-mon[36589]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:33:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:56 smithi145 ceph-mon[32151]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T18:33:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:33:56.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:56 smithi192 ceph-mon[36589]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T18:33:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:33:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:56 smithi164 ceph-mon[36248]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-28T18:33:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:33:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:57 smithi145 ceph-mon[32151]: Deploying daemon osd.9 on smithi145 2024-03-28T18:33:57.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:57 smithi192 ceph-mon[36589]: Deploying daemon osd.9 on smithi145 2024-03-28T18:33:57.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:57 smithi164 ceph-mon[36248]: Deploying daemon osd.9 on smithi145 2024-03-28T18:33:58.593 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:33:58 smithi145 ceph-mon[32151]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:33:58.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:33:58 smithi192 ceph-mon[36589]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:33:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:33:58 smithi164 ceph-mon[36248]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T18:34:00.637 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:00 smithi145 ceph-mon[32151]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T18:34:00.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:00 smithi192 ceph-mon[36589]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T18:34:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:00 smithi164 ceph-mon[36248]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-28T18:34:02.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:02 smithi145 ceph-mon[32151]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:02.725 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:02 smithi192 ceph-mon[36589]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:02 smithi164 ceph-mon[36248]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:03 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:03.903 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.903 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.903 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:03 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:03 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:04.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:04 smithi145 ceph-mon[32151]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:04.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:04 smithi192 ceph-mon[36589]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:04 smithi164 ceph-mon[36248]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:06.588 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.587+0000 7f7e117fa700 1 -- 172.21.15.145:0/3028592414 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7e140fe760 con 0x7f7e000a2890 2024-03-28T18:34:06.588 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 9 on host 'smithi145' 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 msgr2=0x7f7e000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 0x7f7e000a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009a10 tx=0x7f7e04009190 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 msgr2=0x7f7e1419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c9c0 tx=0x7f7dfc00ccd0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 shutdown_connections 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7e000a2890 0x7f7e000a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e14102c60 0x7f7e141a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e141010c0 0x7f7e1419a5a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 --2- 172.21.15.145:0/3028592414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141022c0 0x7f7e1419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 >> 172.21.15.145:0/3028592414 conn(0x7f7e140fc660 msgr2=0x7f7e140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.590+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 shutdown_connections 2024-03-28T18:34:06.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:34:06.591+0000 7f7e1a298700 1 -- 172.21.15.145:0/3028592414 wait complete. 2024-03-28T18:34:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:34:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:06 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:06 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:06.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:34:06.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:06.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:06 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:07.299 DEBUG:teuthology.orchestra.run.smithi145:osd.9> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T18:34:07.302 INFO:tasks.cephadm:Deploying osd.10 on smithi164 with /dev/vg_nvme/lv_1... 2024-03-28T18:34:07.302 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:34:07.552 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:34:07.794 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:07 smithi145 ceph-mon[32151]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:34:07.795 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:07 smithi145 ceph-mon[32151]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:34:07.795 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:07 smithi145 ceph-mon[32151]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:34:07.795 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:07 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:07.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:07 smithi192 ceph-mon[36589]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:34:07.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:07 smithi192 ceph-mon[36589]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:34:07.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:07 smithi192 ceph-mon[36589]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:34:07.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:07 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:07.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:07 smithi164 ceph-mon[36248]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:34:07.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:07 smithi164 ceph-mon[36248]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:34:07.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:07 smithi164 ceph-mon[36248]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-03-28T18:34:07.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:07 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:34:08.065+0000 7f8d91740700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T18:34:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:34:08.077+0000 7f8d88d2a700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:34:08.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:08.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:34:08.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T18:34:08.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:08 smithi192 ceph-mon[36589]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' 2024-03-28T18:34:08.940 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:08.941 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:34:08.941 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T18:34:08.941 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.941 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.941 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:08 smithi164 ceph-mon[36248]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:08.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:08 smithi145 ceph-mon[32151]: from='osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093]' entity='osd.9' 2024-03-28T18:34:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:34:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:34:09.515 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-28T18:34:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-28T18:34:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126889 s, 826 MB/s 2024-03-28T18:34:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] boot 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:34:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] boot 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:34:09.926 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:10.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:34:10.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:34:10.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:10.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: osd.9 [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] boot 2024-03-28T18:34:10.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:34:10.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:34:10.271 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_1 2024-03-28T18:34:10.515 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:34:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T18:34:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:10.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:10.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:34:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T18:34:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:10.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:11.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:11 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi145 2024-03-28T18:34:11.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:11 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi145 to 4272M 2024-03-28T18:34:11.929 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:11 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi145 2024-03-28T18:34:11.930 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:11 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi145 to 4272M 2024-03-28T18:34:12.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:11 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi145 2024-03-28T18:34:12.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:11 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi145 to 4272M 2024-03-28T18:34:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 -- 172.21.15.164:0/3550743487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d4102c60 msgr2=0x7f11d4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 --2- 172.21.15.164:0/3550743487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d4102c60 0x7f11d4109510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00b0d0 tx=0x7f11cc00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 -- 172.21.15.164:0/3550743487 shutdown_connections 2024-03-28T18:34:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 --2- 172.21.15.164:0/3550743487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d4102c60 0x7f11d4109510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 --2- 172.21.15.164:0/3550743487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 0x7f11d41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 --2- 172.21.15.164:0/3550743487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d41022c0 0x7f11d4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 -- 172.21.15.164:0/3550743487 >> 172.21.15.164:0/3550743487 conn(0x7f11d40fc660 msgr2=0x7f11d40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 -- 172.21.15.164:0/3550743487 shutdown_connections 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.150+0000 7f11db42f700 1 -- 172.21.15.164:0/3550743487 wait complete. 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 Processor -- start 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 -- start start 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 0x7f11d406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 0x7f11d419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d410ccb0 con 0x7f11d4102c60 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d410cb30 con 0x7f11d41022c0 2024-03-28T18:34:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11db42f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d410ce30 con 0x7f11d41010c0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11d89ca700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11d89ca700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48840/0 (socket says 172.21.15.164:48840) 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11d89ca700 1 -- 172.21.15.164:0/3715019758 learned_addr learned my addr 172.21.15.164:0/3715019758 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11d91cb700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 0x7f11d406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.151+0000 7f11d99cc700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 0x7f11d419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 -- 172.21.15.164:0/3715019758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 msgr2=0x7f11d406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 0x7f11d406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 -- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 msgr2=0x7f11d419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 0x7f11d419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 -- 172.21.15.164:0/3715019758 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d41a0050 con 0x7f11d41022c0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d99cc700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 0x7f11d419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11d89ca700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d950 tx=0x7f11c400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4016070 con 0x7f11d41022c0 2024-03-28T18:34:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d41a0340 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11c4009e20 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c400f620 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.152+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d41a0880 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.153+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4068e50 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.154+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f11c4019070 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.155+0000 7f11ca7fc700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 0x7f11c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.155+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f11c40cde70 con 0x7f11d41022c0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.155+0000 7f11d91cb700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 0x7f11c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.155+0000 7f11d91cb700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 0x7f11c00a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11d4102120 tx=0x7f11d000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.158+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f11c409b710 con 0x7f11d41022c0 2024-03-28T18:34:12.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:12.365+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f11d4062690 con 0x7f11c00a2890 2024-03-28T18:34:12.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:12 smithi192 ceph-mon[36589]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:12.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:12.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:12.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:12 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:12 smithi164 ceph-mon[36248]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:12 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:12 smithi145 ceph-mon[32151]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:12 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:13 smithi192 ceph-mon[36589]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:13.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:13 smithi164 ceph-mon[36248]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:13.988 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:13 smithi145 ceph-mon[32151]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:14 smithi164 ceph-mon[36248]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:14 smithi145 ceph-mon[32151]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:15.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:14 smithi192 ceph-mon[36589]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/1447066685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]': finished 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:15.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:15 smithi164 ceph-mon[36248]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/1447066685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]': finished 2024-03-28T18:34:16.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:34:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:16.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:15 smithi145 ceph-mon[32151]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:16.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/1447066685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:16.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]: dispatch 2024-03-28T18:34:16.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bce4fe04-fb9d-4108-ab8f-420cae8243ed"}]': finished 2024-03-28T18:34:16.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:34:16.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:16.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:15 smithi192 ceph-mon[36589]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:16 smithi164 ceph-mon[36248]: from='client.? 172.21.15.164:0/4273669783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:17.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:16 smithi145 ceph-mon[32151]: from='client.? 172.21.15.164:0/4273669783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:17.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:16 smithi192 ceph-mon[36589]: from='client.? 172.21.15.164:0/4273669783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:17.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:17 smithi164 ceph-mon[36248]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:18.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:17 smithi145 ceph-mon[32151]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:18.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:17 smithi192 ceph-mon[36589]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:20.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:20 smithi145 ceph-mon[32151]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:20 smithi192 ceph-mon[36589]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:20.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:20 smithi164 ceph-mon[36248]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:22.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:22 smithi145 ceph-mon[32151]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:22.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:22 smithi192 ceph-mon[36589]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:22.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:22 smithi164 ceph-mon[36248]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:23.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:23 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:34:23.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:23 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:23.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:23 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:34:23.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:23 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:23.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:23 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:34:23.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:23 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:24.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:24 smithi164 ceph-mon[36248]: Deploying daemon osd.10 on smithi164 2024-03-28T18:34:24.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:24 smithi164 ceph-mon[36248]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:24 smithi145 ceph-mon[32151]: Deploying daemon osd.10 on smithi164 2024-03-28T18:34:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:24 smithi145 ceph-mon[32151]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:24 smithi192 ceph-mon[36589]: Deploying daemon osd.10 on smithi164 2024-03-28T18:34:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:24 smithi192 ceph-mon[36589]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:26.822 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:26 smithi164 ceph-mon[36248]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:26.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:26 smithi192 ceph-mon[36589]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:27.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:26 smithi145 ceph-mon[32151]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:28 smithi164 ceph-mon[36248]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:28 smithi192 ceph-mon[36589]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:29.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:28 smithi145 ceph-mon[32151]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:29.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:29.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:29.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:30.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:30 smithi164 ceph-mon[36248]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:30 smithi192 ceph-mon[36589]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:31.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:30 smithi145 ceph-mon[32151]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:31.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.782+0000 7f11ca7fc700 1 -- 172.21.15.164:0/3715019758 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f11d4062690 con 0x7f11c00a2890 2024-03-28T18:34:31.783 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 10 on host 'smithi164' 2024-03-28T18:34:31.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 msgr2=0x7f11c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:31.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 0x7f11c00a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11d4102120 tx=0x7f11d000b410 comp rx=0 tx=0).stop 2024-03-28T18:34:31.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 msgr2=0x7f11d419d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11c400d950 tx=0x7f11c400dc60 comp rx=0 tx=0).stop 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 shutdown_connections 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f11c00a2890 0x7f11c00a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11d4102c60 0x7f11d419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11d41010c0 0x7f11d406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 --2- 172.21.15.164:0/3715019758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11d41022c0 0x7f11d419d130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 >> 172.21.15.164:0/3715019758 conn(0x7f11d40fc660 msgr2=0x7f11d41055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 shutdown_connections 2024-03-28T18:34:31.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-28T18:34:31.784+0000 7f11db42f700 1 -- 172.21.15.164:0/3715019758 wait complete. 2024-03-28T18:34:32.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:32 smithi164 ceph-mon[36248]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:32.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:32 smithi164 ceph-mon[36248]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:32.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:32 smithi164 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:32.778 DEBUG:teuthology.orchestra.run.smithi164:osd.10> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T18:34:32.780 INFO:tasks.cephadm:Deploying osd.11 on smithi192 with /dev/vg_nvme/lv_1... 2024-03-28T18:34:32.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:34:32.810 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:32 smithi192 ceph-mon[36589]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:32.810 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.810 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.811 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:32 smithi192 ceph-mon[36589]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:32.811 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:32 smithi192 ceph-mon[36589]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:32.829 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:32 smithi145 ceph-mon[32151]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:32.829 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.829 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:32.829 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:32 smithi145 ceph-mon[32151]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:32.829 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:32 smithi145 ceph-mon[32151]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:34:33.028 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:34:33.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:33.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:33 smithi164 ceph-mon[36248]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:34.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:34.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:33 smithi145 ceph-mon[32151]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:34.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:33 smithi192 ceph-mon[36589]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:34.462 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:34:34.177+0000 7f94d7282700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T18:34:34.463 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:34:34.190+0000 7f94ce86c700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:34:34.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:34:34.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-mon[36248]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T18:34:34.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:34 smithi164 ceph-mon[36248]: from='osd.10 ' entity='osd.10' 2024-03-28T18:34:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:34 smithi192 ceph-mon[36589]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:34:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:34 smithi192 ceph-mon[36589]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T18:34:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:34 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:34 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:34.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:34 smithi192 ceph-mon[36589]: from='osd.10 ' entity='osd.10' 2024-03-28T18:34:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:34 smithi145 ceph-mon[32151]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-28T18:34:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:34 smithi145 ceph-mon[32151]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T18:34:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:34 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:34 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:35.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:34 smithi145 ceph-mon[32151]: from='osd.10 ' entity='osd.10' 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114584 s, 915 MB/s 2024-03-28T18:34:35.101 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-28T18:34:35.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] boot 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:35 smithi145 ceph-mon[32151]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:36.093 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:34:36.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] boot 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:35 smithi192 ceph-mon[36589]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: osd.10 [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] boot 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:35 smithi164 ceph-mon[36248]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:34:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:34:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi164 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi164 to 4270M 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:36.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi164 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi164 to 4270M 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi164 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi164 to 4270M 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:37.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.849+0000 7f44448e1700 1 -- 172.21.15.192:0/158499505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 msgr2=0x7f44401014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.849+0000 7f44448e1700 1 --2- 172.21.15.192:0/158499505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f44401014e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4430009990 tx=0x7f4430009ca0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 -- 172.21.15.192:0/158499505 shutdown_connections 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 --2- 172.21.15.192:0/158499505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4440102c60 0x7f4440109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 --2- 172.21.15.192:0/158499505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f44401014e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 --2- 172.21.15.192:0/158499505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44401022c0 0x7f4440102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 -- 172.21.15.192:0/158499505 >> 172.21.15.192:0/158499505 conn(0x7f44400fc680 msgr2=0x7f44400feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 -- 172.21.15.192:0/158499505 shutdown_connections 2024-03-28T18:34:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 -- 172.21.15.192:0/158499505 wait complete. 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 Processor -- start 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.850+0000 7f44448e1700 1 -- start start 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44401022c0 0x7f444019d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 0x7f444019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010cd10 con 0x7f4440102c60 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010cb90 con 0x7f44401022c0 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010ce90 con 0x7f44401010c0 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:43194/0 (socket says 172.21.15.192:43194) 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 -- 172.21.15.192:0/2168199034 learned_addr learned my addr 172.21.15.192:0/2168199034 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443dd9b700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44401022c0 0x7f444019d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443ed9d700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 0x7f444019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 -- 172.21.15.192:0/2168199034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44401022c0 msgr2=0x7f444019d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44401022c0 0x7f444019d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 -- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 msgr2=0x7f444019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 0x7f444019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 -- 172.21.15.192:0/2168199034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44401a00a0 con 0x7f44401010c0 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443ed9d700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 0x7f444019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f443e59c700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4430000c00 tx=0x7f44300047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430003d30 con 0x7f44401010c0 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44401a0330 con 0x7f44401010c0 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4430004830 con 0x7f44401010c0 2024-03-28T18:34:37.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44401a0800 con 0x7f44401010c0 2024-03-28T18:34:37.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.851+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430005580 con 0x7f44401010c0 2024-03-28T18:34:37.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.852+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440068e50 con 0x7f44401010c0 2024-03-28T18:34:37.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.853+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4430003ed0 con 0x7f44401010c0 2024-03-28T18:34:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.854+0000 7f442f7fe700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 0x7f44280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.854+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f44300cfa60 con 0x7f44401010c0 2024-03-28T18:34:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.854+0000 7f443dd9b700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 0x7f44280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.855+0000 7f443dd9b700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 0x7f44280a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4434005950 tx=0x7f443400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:37.858 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:37.858+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4430013070 con 0x7f44401010c0 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:37 smithi192 ceph-mon[36589]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:38.060 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:38.060+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44400625d0 con 0x7f44280a2890 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:37 smithi164 ceph-mon[36248]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:38.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:37 smithi145 ceph-mon[32151]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:38.958 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:38 smithi192 ceph-mon[36589]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:34:38.958 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:38 smithi192 ceph-mon[36589]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:38.958 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:38.958 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:38.958 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:38 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:38 smithi164 ceph-mon[36248]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:34:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:38 smithi164 ceph-mon[36248]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:39.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:38 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:38 smithi145 ceph-mon[32151]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:34:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:38 smithi145 ceph-mon[32151]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:38 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:39 smithi192 ceph-mon[36589]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T18:34:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:39 smithi192 ceph-mon[36589]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:34:40.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:39 smithi164 ceph-mon[36248]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T18:34:40.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:39 smithi164 ceph-mon[36248]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:34:40.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:39 smithi145 ceph-mon[32151]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T18:34:40.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:39 smithi145 ceph-mon[32151]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T18:34:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:40 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/771155280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:40 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:40 smithi192 ceph-mon[36589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]': finished 2024-03-28T18:34:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:40 smithi192 ceph-mon[36589]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:34:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:40 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:40 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/771155280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:40 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:40 smithi164 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]': finished 2024-03-28T18:34:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:40 smithi164 ceph-mon[36248]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:34:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:40 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:40 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/771155280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:40 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]: dispatch 2024-03-28T18:34:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:40 smithi145 ceph-mon[32151]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95d0633-327c-4fe9-a150-07eaa70cbcec"}]': finished 2024-03-28T18:34:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:40 smithi145 ceph-mon[32151]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:34:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:40 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:42.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:41 smithi164 ceph-mon[36248]: from='client.? 172.21.15.192:0/463534425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:42.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:41 smithi164 ceph-mon[36248]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-28T18:34:42.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:41 smithi192 ceph-mon[36589]: from='client.? 172.21.15.192:0/463534425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:42.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:41 smithi192 ceph-mon[36589]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-28T18:34:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:41 smithi145 ceph-mon[32151]: from='client.? 172.21.15.192:0/463534425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:34:42.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:41 smithi145 ceph-mon[32151]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-28T18:34:44.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:44 smithi164 ceph-mon[36248]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:34:44.770 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:44 smithi192 ceph-mon[36589]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:34:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:44 smithi145 ceph-mon[32151]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:34:46.750 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:46 smithi192 ceph-mon[36589]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-28T18:34:46.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:46 smithi145 ceph-mon[32151]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-28T18:34:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:46 smithi164 ceph-mon[36248]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-28T18:34:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:48 smithi145 ceph-mon[32151]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-28T18:34:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:34:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:48.780 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:48 smithi192 ceph-mon[36589]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-28T18:34:48.780 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:34:48.780 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:48 smithi164 ceph-mon[36248]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-28T18:34:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:34:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:49.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:49 smithi192 ceph-mon[36589]: Deploying daemon osd.11 on smithi192 2024-03-28T18:34:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:49 smithi145 ceph-mon[32151]: Deploying daemon osd.11 on smithi192 2024-03-28T18:34:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:49 smithi164 ceph-mon[36248]: Deploying daemon osd.11 on smithi192 2024-03-28T18:34:50.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:50 smithi145 ceph-mon[32151]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:50.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:50 smithi192 ceph-mon[36589]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:50 smithi164 ceph-mon[36248]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:52.820 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:52 smithi192 ceph-mon[36589]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:52 smithi164 ceph-mon[36248]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:52 smithi145 ceph-mon[32151]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:54 smithi192 ceph-mon[36589]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:54.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:54.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:54 smithi164 ceph-mon[36248]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:54.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:54 smithi145 ceph-mon[32151]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:55.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:56.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:56 smithi192 ceph-mon[36589]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:56 smithi164 ceph-mon[36248]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:57.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:56 smithi145 ceph-mon[32151]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:57.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.777+0000 7f442f7fe700 1 -- 172.21.15.192:0/2168199034 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f44400625d0 con 0x7f44280a2890 2024-03-28T18:34:57.779 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 11 on host 'smithi192' 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 msgr2=0x7f44280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 0x7f44280a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4434005950 tx=0x7f443400a300 comp rx=0 tx=0).stop 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 msgr2=0x7f444006b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4430000c00 tx=0x7f44300047b0 comp rx=0 tx=0).stop 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 shutdown_connections 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44280a2890 0x7f44280a4d50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4440102c60 0x7f444019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:57.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44401010c0 0x7f444006b370 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:57.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 --2- 172.21.15.192:0/2168199034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44401022c0 0x7f444019d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:57.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 >> 172.21.15.192:0/2168199034 conn(0x7f44400fc680 msgr2=0x7f44401055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:57.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 shutdown_connections 2024-03-28T18:34:57.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:34:57.779+0000 7f44448e1700 1 -- 172.21.15.192:0/2168199034 wait complete. 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:34:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:34:58.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.594 DEBUG:teuthology.orchestra.run.smithi192:osd.11> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T18:34:58.596 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T18:34:58.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T18:34:58.849 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:34:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-mon[36589]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:34:58.908 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:34:58.760+0000 7fe055ada700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T18:34:58.908 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:34:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:34:58.773+0000 7fe0518cd700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:34:58.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:58.970 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:34:58.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:34:58.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:58.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:58.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:34:58.971 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:58 smithi145 ceph-mon[32151]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:58 smithi164 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-28T18:35:00.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:59 smithi145 ceph-mon[32151]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:35:00.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:59 smithi145 ceph-mon[32151]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T18:35:00.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:59 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:59 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:34:59 smithi145 ceph-mon[32151]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:35:00.030 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:59 smithi192 ceph-mon[36589]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:35:00.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:59 smithi192 ceph-mon[36589]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T18:35:00.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:59 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:59 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:34:59 smithi192 ceph-mon[36589]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:35:00.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:59 smithi164 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-28T18:35:00.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:59 smithi164 ceph-mon[36248]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T18:35:00.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:59 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:59 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:34:59 smithi164 ceph-mon[36248]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:35:00.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 -- 172.21.15.145:0/595378118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548102c40 msgr2=0x7fe548105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:00.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/595378118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548102c40 0x7fe548105030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe538009990 tx=0x7fe538009ca0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 -- 172.21.15.145:0/595378118 shutdown_connections 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/595378118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548107ea0 0x7fe54810a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/595378118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548102c40 0x7fe548105030 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/595378118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548105570 0x7fe548107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 -- 172.21.15.145:0/595378118 >> 172.21.15.145:0/595378118 conn(0x7fe5480fc690 msgr2=0x7fe5480feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 -- 172.21.15.145:0/595378118 shutdown_connections 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 -- 172.21.15.145:0/595378118 wait complete. 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.668+0000 7fe54cfc4700 1 Processor -- start 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 -- start start 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548102c40 0x7fe548198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:00.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 0x7fe54819f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810ddc0 con 0x7fe548107ea0 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810dc40 con 0x7fe548102c40 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54cfc4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54810df40 con 0x7fe548105570 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe546d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 0x7fe54819f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56010/0 (socket says 172.21.15.145:56010) 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 -- 172.21.15.145:0/2508372292 learned_addr learned my addr 172.21.15.145:0/2508372292 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe546d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 0x7fe54819f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36432/0 (socket says 172.21.15.145:36432) 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe54659c700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548102c40 0x7fe548198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 -- 172.21.15.145:0/2508372292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548102c40 msgr2=0x7fe548198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:00.672 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548102c40 0x7fe548198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 -- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 msgr2=0x7fe54819f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 0x7fe54819f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.669+0000 7fe53dd9b700 1 -- 172.21.15.145:0/2508372292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54819fff0 con 0x7fe548105570 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe53dd9b700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe53000d9b0 tx=0x7fe53000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530016070 con 0x7fe548105570 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5481a02e0 con 0x7fe548105570 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe530004d10 con 0x7fe548105570 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530005530 con 0x7fe548105570 2024-03-28T18:35:00.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.670+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5481a0870 con 0x7fe548105570 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.671+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe548068e50 con 0x7fe548105570 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.672+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe530004750 con 0x7fe548105570 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.672+0000 7fe53ffff700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 0x7fe5340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.672+0000 7fe54659c700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 0x7fe5340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.672+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5300cea90 con 0x7fe548105570 2024-03-28T18:35:00.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.673+0000 7fe54659c700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 0x7fe5340a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe53800b2e0 tx=0x7fe538005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:00.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.676+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe53001f020 con 0x7fe548105570 2024-03-28T18:35:00.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.873+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe54802da30 con 0x7fe548105570 2024-03-28T18:35:00.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:00 smithi145 ceph-mon[32151]: purged_snaps scrub starts 2024-03-28T18:35:00.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:00 smithi145 ceph-mon[32151]: purged_snaps scrub ok 2024-03-28T18:35:00.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:00 smithi145 ceph-mon[32151]: osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] boot 2024-03-28T18:35:00.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:00 smithi145 ceph-mon[32151]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:35:00.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:00.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.874+0000 7fe53ffff700 1 -- 172.21.15.145:0/2508372292 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe53009bf30 con 0x7fe548105570 2024-03-28T18:35:00.875 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:00.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 msgr2=0x7fe5340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:00.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 0x7fe5340a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe53800b2e0 tx=0x7fe538005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 msgr2=0x7fe5481988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe53000d9b0 tx=0x7fe53000dcc0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 shutdown_connections 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe5340a2890 0x7fe5340a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe548107ea0 0x7fe54819f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe548102c40 0x7fe548198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 --2- 172.21.15.145:0/2508372292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe548105570 0x7fe5481988d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 >> 172.21.15.145:0/2508372292 conn(0x7fe5480fc690 msgr2=0x7fe5481036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 shutdown_connections 2024-03-28T18:35:00.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:00.876+0000 7fe54cfc4700 1 -- 172.21.15.145:0/2508372292 wait complete. 2024-03-28T18:35:01.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:00 smithi192 ceph-mon[36589]: purged_snaps scrub starts 2024-03-28T18:35:01.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:00 smithi192 ceph-mon[36589]: purged_snaps scrub ok 2024-03-28T18:35:01.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:00 smithi192 ceph-mon[36589]: osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] boot 2024-03-28T18:35:01.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:00 smithi192 ceph-mon[36589]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:35:01.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:00 smithi164 ceph-mon[36248]: purged_snaps scrub starts 2024-03-28T18:35:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:00 smithi164 ceph-mon[36248]: purged_snaps scrub ok 2024-03-28T18:35:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:00 smithi164 ceph-mon[36248]: osd.11 [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] boot 2024-03-28T18:35:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:00 smithi164 ceph-mon[36248]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:35:01.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:35:01.632 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1711650899,"num_in_osds":12,"osd_in_since":1711650880,"num_remapped_pgs":0} 2024-03-28T18:35:01.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2508372292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: Detected new or changed devices on smithi192 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:01.833 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-28T18:35:01.834 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:01.834 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:01.834 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:01.834 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:01 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:01.878 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2508372292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: Detected new or changed devices on smithi192 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:02.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:01 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2508372292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: Detected new or changed devices on smithi192 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:01 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 -- 172.21.15.145:0/486982948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 msgr2=0x7fbcdc1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 --2- 172.21.15.145:0/486982948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1085c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0099c0 tx=0x7fbccc009cd0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 -- 172.21.15.145:0/486982948 shutdown_connections 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 --2- 172.21.15.145:0/486982948 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 --2- 172.21.15.145:0/486982948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1085c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 --2- 172.21.15.145:0/486982948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 -- 172.21.15.145:0/486982948 >> 172.21.15.145:0/486982948 conn(0x7fbcdc077fc0 msgr2=0x7fbcdc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 -- 172.21.15.145:0/486982948 shutdown_connections 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.603+0000 7fbce1103700 1 -- 172.21.15.145:0/486982948 wait complete. 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 Processor -- start 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 -- start start 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc112420 con 0x7fbcdc1081e0 2024-03-28T18:35:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1122a0 con 0x7fbcdc06a7e0 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.604+0000 7fbce1103700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1125a0 con 0x7fbcdc069e40 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56164/0 (socket says 172.21.15.145:56164) 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 -- 172.21.15.145:0/1897593378 learned_addr learned my addr 172.21.15.145:0/1897593378 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcda59c700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 -- 172.21.15.145:0/1897593378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 msgr2=0x7fbcdc19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 -- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 msgr2=0x7fbcdc1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 -- 172.21.15.145:0/1897593378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc198fa0 con 0x7fbcdc069e40 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcda59c700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdb59e700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcdad9d700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbccc006010 tx=0x7fbccc003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc009e60 con 0x7fbcdc069e40 2024-03-28T18:35:03.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc199230 con 0x7fbcdc069e40 2024-03-28T18:35:03.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbccc003680 con 0x7fbcdc069e40 2024-03-28T18:35:03.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0173f0 con 0x7fbcdc069e40 2024-03-28T18:35:03.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.605+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc1a53a0 con 0x7fbcdc069e40 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.606+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc103890 con 0x7fbcdc069e40 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.607+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbccc00f660 con 0x7fbcdc069e40 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.608+0000 7fbcc3fff700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 0x7fbcc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.608+0000 7fbcda59c700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 0x7fbcc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.608+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbccc0cf330 con 0x7fbcdc069e40 2024-03-28T18:35:03.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.608+0000 7fbcda59c700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 0x7fbcc40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc199e80 tx=0x7fbcd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:03.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.611+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbccc013070 con 0x7fbcdc069e40 2024-03-28T18:35:03.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.810+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcdc02da30 con 0x7fbcdc069e40 2024-03-28T18:35:03.811 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.811+0000 7fbcc3fff700 1 -- 172.21.15.145:0/1897593378 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fbccc09c900 con 0x7fbcdc069e40 2024-03-28T18:35:03.811 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:03.811 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":83,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:00.757071+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:03.813 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 msgr2=0x7fbcc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 0x7fbcc40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc199e80 tx=0x7fbcd0009260 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 msgr2=0x7fbcdc19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbccc006010 tx=0x7fbccc003e00 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 shutdown_connections 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbcc40a2890 0x7fbcc40a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc1081e0 0x7fbcdc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcdc069e40 0x7fbcdc19e5f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 --2- 172.21.15.145:0/1897593378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcdc06a7e0 0x7fbcdc19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.813+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 >> 172.21.15.145:0/1897593378 conn(0x7fbcdc077fc0 msgr2=0x7fbcdc0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.814+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 shutdown_connections 2024-03-28T18:35:03.814 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:03.814+0000 7fbce1103700 1 -- 172.21.15.145:0/1897593378 wait complete. 2024-03-28T18:35:04.578 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:35:04.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T18:35:04.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:04 smithi192 ceph-mon[36589]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:04.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1897593378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:04 smithi145 ceph-mon[32151]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1897593378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:04.819 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:35:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:04 smithi164 ceph-mon[36248]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1897593378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:06.501 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 -- 172.21.15.145:0/3114856699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40101070 msgr2=0x7f9c401014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:06.501 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 --2- 172.21.15.145:0/3114856699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40101070 0x7f9c401014f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009990 tx=0x7f9c30009ca0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.501 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 -- 172.21.15.145:0/3114856699 shutdown_connections 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 --2- 172.21.15.145:0/3114856699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 0x7f9c4010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 --2- 172.21.15.145:0/3114856699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40107120 0x7f9c40107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 --2- 172.21.15.145:0/3114856699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40101070 0x7f9c401014f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.499+0000 7f9c479f7700 1 -- 172.21.15.145:0/3114856699 >> 172.21.15.145:0/3114856699 conn(0x7f9c400780a0 msgr2=0x7f9c400784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- 172.21.15.145:0/3114856699 shutdown_connections 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- 172.21.15.145:0/3114856699 wait complete. 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 Processor -- start 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- start start 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40101070 0x7f9c4010e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 0x7f9c4010eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40112530 con 0x7f9c40107120 2024-03-28T18:35:06.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c401123b0 con 0x7f9c40101a30 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.500+0000 7f9c479f7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c401126b0 con 0x7f9c40101070 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50508/0 (socket says 172.21.15.145:50508) 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c44f92700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 0x7f9c4010eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 -- 172.21.15.145:0/2850986608 learned_addr learned my addr 172.21.15.145:0/2850986608 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45793700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40101070 0x7f9c4010e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 -- 172.21.15.145:0/2850986608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40101070 msgr2=0x7f9c4010e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40101070 0x7f9c4010e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 -- 172.21.15.145:0/2850986608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 msgr2=0x7f9c4010eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 0x7f9c4010eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 -- 172.21.15.145:0/2850986608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c401a6f50 con 0x7f9c40107120 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c45f94700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b970 tx=0x7f9c3800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38019070 con 0x7f9c40107120 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c401a7240 con 0x7f9c40107120 2024-03-28T18:35:06.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c3800ce10 con 0x7f9c40107120 2024-03-28T18:35:06.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c401a7750 con 0x7f9c40107120 2024-03-28T18:35:06.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.501+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38007370 con 0x7f9c40107120 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.502+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c40102760 con 0x7f9c40107120 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.503+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9c3801c070 con 0x7f9c40107120 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.504+0000 7f9c367fc700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 0x7f9c2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.504+0000 7f9c45793700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 0x7f9c2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.504+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9c380cd970 con 0x7f9c40107120 2024-03-28T18:35:06.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.504+0000 7f9c45793700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 0x7f9c2c0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c006fd0 tx=0x7f9c3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:06.508 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.508+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c3809aeb0 con 0x7f9c40107120 2024-03-28T18:35:06.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:06 smithi145 ceph-mon[32151]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:06.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.704+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c4002da30 con 0x7f9c40107120 2024-03-28T18:35:06.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.704+0000 7f9c367fc700 1 -- 172.21.15.145:0/2850986608 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c3809a7b0 con 0x7f9c40107120 2024-03-28T18:35:06.705 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:35:06.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.706+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 msgr2=0x7f9c2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:06.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.706+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 0x7f9c2c0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c006fd0 tx=0x7f9c3c008040 comp rx=0 tx=0).stop 2024-03-28T18:35:06.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.706+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 msgr2=0x7f9c4010f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:06.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.706+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b970 tx=0x7f9c3800bc80 comp rx=0 tx=0).stop 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 shutdown_connections 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9c2c0a2890 0x7f9c2c0a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c40107120 0x7f9c4010f3e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c40101070 0x7f9c4010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 --2- 172.21.15.145:0/2850986608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c40101a30 0x7f9c4010eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 >> 172.21.15.145:0/2850986608 conn(0x7f9c400780a0 msgr2=0x7f9c400ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 shutdown_connections 2024-03-28T18:35:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:06.707+0000 7f9c479f7700 1 -- 172.21.15.145:0/2850986608 wait complete. 2024-03-28T18:35:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:06 smithi192 ceph-mon[36589]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:06 smithi164 ceph-mon[36248]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.a on smithi145 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.d on smithi145 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.g on smithi145 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.j on smithi145 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.b on smithi164 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.e on smithi164 2024-03-28T18:35:07.382 INFO:tasks.cephadm:Adding mds.h on smithi164 2024-03-28T18:35:07.383 INFO:tasks.cephadm:Adding mds.k on smithi164 2024-03-28T18:35:07.383 INFO:tasks.cephadm:Adding mds.c on smithi192 2024-03-28T18:35:07.383 INFO:tasks.cephadm:Adding mds.f on smithi192 2024-03-28T18:35:07.383 INFO:tasks.cephadm:Adding mds.i on smithi192 2024-03-28T18:35:07.383 INFO:tasks.cephadm:Adding mds.l on smithi192 2024-03-28T18:35:07.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l' 2024-03-28T18:35:07.633 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:35:07.755 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:07 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2850986608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:07.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:07 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2850986608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:07.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:07 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2850986608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:08.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:08 smithi192 ceph-mon[36589]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:08 smithi164 ceph-mon[36248]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:09.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:08 smithi145 ceph-mon[32151]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 -- 172.21.15.192:0/3296052495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 msgr2=0x7f4e98101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/3296052495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e98101510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009990 tx=0x7f4e94009ca0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 -- 172.21.15.192:0/3296052495 shutdown_connections 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/3296052495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 0x7f4e98109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/3296052495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e98101510 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/3296052495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 0x7f4e98102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.250 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 -- 172.21.15.192:0/3296052495 >> 172.21.15.192:0/3296052495 conn(0x7f4e980fc6b0 msgr2=0x7f4e980fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 -- 172.21.15.192:0/3296052495 shutdown_connections 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 -- 172.21.15.192:0/3296052495 wait complete. 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.248+0000 7f4e9fbba700 1 Processor -- start 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 -- start start 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 0x7f4e9819d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 0x7f4e9819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810cce0 con 0x7f4e98102c90 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810cb60 con 0x7f4e981022f0 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9fbba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810ce60 con 0x7f4e981010f0 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:54810/0 (socket says 172.21.15.192:54810) 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 -- 172.21.15.192:0/4032423756 learned_addr learned my addr 172.21.15.192:0/4032423756 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-28T18:35:09.251 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9e157700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 0x7f4e9819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d155700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 0x7f4e9819d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 -- 172.21.15.192:0/4032423756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 msgr2=0x7f4e9819d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 0x7f4e9819d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 -- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 msgr2=0x7f4e9819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 0x7f4e9819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 -- 172.21.15.192:0/4032423756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e981a00a0 con 0x7f4e981010f0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e9d956700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e940039c0 tx=0x7f4e940047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.249+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94003c70 con 0x7f4e981010f0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.250+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e981a0330 con 0x7f4e981010f0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.250+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e94004830 con 0x7f4e981010f0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.250+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94005440 con 0x7f4e981010f0 2024-03-28T18:35:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.250+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e981a0780 con 0x7f4e981010f0 2024-03-28T18:35:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.250+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e98068e50 con 0x7f4e981010f0 2024-03-28T18:35:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.252+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4e94003e10 con 0x7f4e981010f0 2024-03-28T18:35:09.253 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.252+0000 7f4e8effd700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 0x7f4e840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:09.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.252+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e940cfad0 con 0x7f4e981010f0 2024-03-28T18:35:09.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.252+0000 7f4e9d155700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 0x7f4e840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:09.254 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.253+0000 7f4e9d155700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 0x7f4e840a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e88005950 tx=0x7f4e880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:09.255 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.256+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e94013070 con 0x7f4e981010f0 2024-03-28T18:35:09.463 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.463+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e98062890 con 0x7f4e840a2890 2024-03-28T18:35:09.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.471+0000 7f4e8effd700 1 -- 172.21.15.192:0/4032423756 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f4e98062890 con 0x7f4e840a2890 2024-03-28T18:35:09.472 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mds.all update... 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 msgr2=0x7f4e840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 0x7f4e840a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e88005950 tx=0x7f4e880058e0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 msgr2=0x7f4e9806b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e940039c0 tx=0x7f4e940047b0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 shutdown_connections 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e840a2890 0x7f4e840a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98102c90 0x7f4e9819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981010f0 0x7f4e9806b3d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 --2- 172.21.15.192:0/4032423756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981022f0 0x7f4e9819d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:09.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 >> 172.21.15.192:0/4032423756 conn(0x7f4e980fc6b0 msgr2=0x7f4e98105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:09.475 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 shutdown_connections 2024-03-28T18:35:09.475 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-28T18:35:09.474+0000 7f4e9fbba700 1 -- 172.21.15.192:0/4032423756 wait complete. 2024-03-28T18:35:10.095 DEBUG:teuthology.orchestra.run.smithi145:mds.a> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T18:35:10.098 DEBUG:teuthology.orchestra.run.smithi145:mds.d> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T18:35:10.141 DEBUG:teuthology.orchestra.run.smithi145:mds.g> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T18:35:10.183 DEBUG:teuthology.orchestra.run.smithi145:mds.j> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T18:35:10.227 DEBUG:teuthology.orchestra.run.smithi164:mds.b> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T18:35:10.229 DEBUG:teuthology.orchestra.run.smithi164:mds.e> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T18:35:10.271 DEBUG:teuthology.orchestra.run.smithi164:mds.h> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T18:35:10.314 DEBUG:teuthology.orchestra.run.smithi164:mds.k> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T18:35:10.357 DEBUG:teuthology.orchestra.run.smithi192:mds.c> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T18:35:10.360 DEBUG:teuthology.orchestra.run.smithi192:mds.f> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T18:35:10.403 DEBUG:teuthology.orchestra.run.smithi192:mds.i> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T18:35:10.445 DEBUG:teuthology.orchestra.run.smithi192:mds.l> sudo journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T18:35:10.489 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T18:35:10.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- 172.21.15.145:0/3117038204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c1005c0 msgr2=0x7f6c1c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 --2- 172.21.15.145:0/3117038204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c1009e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009900 tx=0x7f6c0402bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- 172.21.15.145:0/3117038204 shutdown_connections 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 --2- 172.21.15.145:0/3117038204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c102170 0x7f6c1c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 --2- 172.21.15.145:0/3117038204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1017b0 0x7f6c1c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 --2- 172.21.15.145:0/3117038204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c1009e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- 172.21.15.145:0/3117038204 >> 172.21.15.145:0/3117038204 conn(0x7f6c1c0fbb30 msgr2=0x7f6c1c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- 172.21.15.145:0/3117038204 shutdown_connections 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- 172.21.15.145:0/3117038204 wait complete. 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 Processor -- start 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.603+0000 7f6c22309640 1 -- start start 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c1017b0 0x7f6c1c196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c10c290 con 0x7f6c1c102170 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c10c110 con 0x7f6c1c1017b0 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c10c410 con 0x7f6c1c1005c0 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50520/0 (socket says 172.21.15.145:50520) 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 -- 172.21.15.145:0/2969551417 learned_addr learned my addr 172.21.15.145:0/2969551417 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c1bfff640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c1b7fe640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c1017b0 0x7f6c1c196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 -- 172.21.15.145:0/2969551417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1005c0 msgr2=0x7f6c1c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 -- 172.21.15.145:0/2969551417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c1017b0 msgr2=0x7f6c1c196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c1017b0 0x7f6c1c196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 -- 172.21.15.145:0/2969551417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c19e580 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c208a6640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400eaa0 tx=0x7f6c1400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14004d20 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c19e870 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c14005020 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140116d0 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.604+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c19ed70 con 0x7f6c1c102170 2024-03-28T18:35:10.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.605+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c1078e0 con 0x7f6c1c102170 2024-03-28T18:35:10.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.607+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6c140051d0 con 0x7f6c1c102170 2024-03-28T18:35:10.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.607+0000 7f6c197fa640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 0x7f6c080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.607+0000 7f6c1bfff640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 0x7f6c080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.607+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c140ceb60 con 0x7f6c1c102170 2024-03-28T18:35:10.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.608+0000 7f6c1bfff640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 0x7f6c080a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000c00 tx=0x7f6c04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:10.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.611+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c14016030 con 0x7f6c1c102170 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: Saving service mds.all spec with placement smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:10.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:10 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: Saving service mds.all spec with placement smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:10.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:10 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: Saving service mds.all spec with placement smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:10.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:10 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.787+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c1c0622a0 con 0x7f6c1c102170 2024-03-28T18:35:10.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.788+0000 7f6c197fa640 1 -- 172.21.15.145:0/2969551417 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f6c1409c0a0 con 0x7f6c1c102170 2024-03-28T18:35:10.788 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:10.788 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":83,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:00.757071+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 msgr2=0x7f6c080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 0x7f6c080a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000c00 tx=0x7f6c04005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 msgr2=0x7f6c1c19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400eaa0 tx=0x7f6c1400eea0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 shutdown_connections 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c1c102170 0x7f6c1c19df10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c1c1017b0 0x7f6c1c196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c080a2800 0x7f6c080a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 --2- 172.21.15.145:0/2969551417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c1005c0 0x7f6c1c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 >> 172.21.15.145:0/2969551417 conn(0x7f6c1c0fbb30 msgr2=0x7f6c1c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:10.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 shutdown_connections 2024-03-28T18:35:10.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.790+0000 7f6c22309640 1 -- 172.21.15.145:0/2969551417 wait complete. 2024-03-28T18:35:10.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:35:10.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 -- 172.21.15.145:0/2057256431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb454104a70 msgr2=0x7fb454106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 --2- 172.21.15.145:0/2057256431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb454104a70 0x7fb454106e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb444009900 tx=0x7fb44402bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 -- 172.21.15.145:0/2057256431 shutdown_connections 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 --2- 172.21.15.145:0/2057256431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4541073a0 0x7fb4541097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 --2- 172.21.15.145:0/2057256431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb454104a70 0x7fb454106e60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 --2- 172.21.15.145:0/2057256431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454102140 0x7fb454104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 -- 172.21.15.145:0/2057256431 >> 172.21.15.145:0/2057256431 conn(0x7fb4540fbb50 msgr2=0x7fb4540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 -- 172.21.15.145:0/2057256431 shutdown_connections 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 -- 172.21.15.145:0/2057256431 wait complete. 2024-03-28T18:35:10.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.906+0000 7fb459e97640 1 Processor -- start 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- start start 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb454102140 0x7fb454199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454104a70 0x7fb45419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45410d3b0 con 0x7fb4541073a0 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45410d230 con 0x7fb454102140 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45410d530 con 0x7fb454104a70 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50552/0 (socket says 172.21.15.145:50552) 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 -- 172.21.15.145:0/2301889017 learned_addr learned my addr 172.21.15.145:0/2301889017 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb4537fe640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb454102140 0x7fb454199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb452ffd640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454104a70 0x7fb45419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 -- 172.21.15.145:0/2301889017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454104a70 msgr2=0x7fb45419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454104a70 0x7fb45419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 -- 172.21.15.145:0/2301889017 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb454102140 msgr2=0x7fb454199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb454102140 0x7fb454199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 -- 172.21.15.145:0/2301889017 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4541a1900 con 0x7fb4541073a0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb453fff640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b970 tx=0x7fb44c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c019070 con 0x7fb4541073a0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb44c00cb40 con 0x7fb4541073a0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c007650 con 0x7fb4541073a0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4541a1bf0 con 0x7fb4541073a0 2024-03-28T18:35:10.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.907+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4541a20f0 con 0x7fb4541073a0 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.909+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb454108e90 con 0x7fb4541073a0 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.910+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb44c01c070 con 0x7fb4541073a0 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.910+0000 7fb450ff9640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 0x7fb43c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.910+0000 7fb4537fe640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 0x7fb43c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.910+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb44c0cdbf0 con 0x7fb4541073a0 2024-03-28T18:35:10.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.911+0000 7fb4537fe640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 0x7fb43c0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb45419b1c0 tx=0x7fb44800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:10.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:10.914+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb44c09aae0 con 0x7fb4541073a0 2024-03-28T18:35:11.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.087+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb45402d300 con 0x7fb4541073a0 2024-03-28T18:35:11.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.088+0000 7fb450ff9640 1 -- 172.21.15.145:0/2301889017 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb44c09eb80 con 0x7fb4541073a0 2024-03-28T18:35:11.088 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 msgr2=0x7fb43c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 0x7fb43c0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb45419b1c0 tx=0x7fb44800b390 comp rx=0 tx=0).stop 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 msgr2=0x7fb4541a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b970 tx=0x7fb44c00be30 comp rx=0 tx=0).stop 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 shutdown_connections 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4541073a0 0x7fb4541a1260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb43c0a2800 0x7fb43c0a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.090 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb454104a70 0x7fb45419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 --2- 172.21.15.145:0/2301889017 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb454102140 0x7fb454199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 >> 172.21.15.145:0/2301889017 conn(0x7fb4540fbb50 msgr2=0x7fb4541018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:11.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 shutdown_connections 2024-03-28T18:35:11.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.090+0000 7fb459e97640 1 -- 172.21.15.145:0/2301889017 wait complete. 2024-03-28T18:35:11.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- 172.21.15.145:0/484172040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1005e0 msgr2=0x7f6f0c100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- 172.21.15.145:0/484172040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c100a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009900 tx=0x7f6f0002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- 172.21.15.145:0/484172040 shutdown_connections 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- 172.21.15.145:0/484172040 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c102190 0x7f6f0c108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- 172.21.15.145:0/484172040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- 172.21.15.145:0/484172040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c100a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- 172.21.15.145:0/484172040 >> 172.21.15.145:0/484172040 conn(0x7f6f0c0fbb50 msgr2=0x7f6f0c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- 172.21.15.145:0/484172040 shutdown_connections 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- 172.21.15.145:0/484172040 wait complete. 2024-03-28T18:35:11.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 Processor -- start 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- start start 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c102190 0x7f6f0c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c10c250 con 0x7f6f0c1017d0 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c10c0d0 con 0x7f6f0c1005e0 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f1315b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c10c3d0 con 0x7f6f0c102190 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.203+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50574/0 (socket says 172.21.15.145:50574) 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 -- 172.21.15.145:0/1130480906 learned_addr learned my addr 172.21.15.145:0/1130480906 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f10ef7640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f116f8640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c102190 0x7f6f0c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 -- 172.21.15.145:0/1130480906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c102190 msgr2=0x7f6f0c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c102190 0x7f6f0c19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 -- 172.21.15.145:0/1130480906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c1005e0 msgr2=0x7f6f0c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 -- 172.21.15.145:0/1130480906 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c19f630 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f0bfff640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800e9f0 tx=0x7f6ef800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800cd80 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ef8005500 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c19f920 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8010690 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.204+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c19fe20 con 0x7f6f0c1017d0 2024-03-28T18:35:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.205+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c1a00b0 con 0x7f6f0c1017d0 2024-03-28T18:35:11.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.206+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6ef80046c0 con 0x7f6f0c1017d0 2024-03-28T18:35:11.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.207+0000 7f6f09ffb640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 0x7f6efc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:11.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.207+0000 7f6f10ef7640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 0x7f6efc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:11.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.207+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ef80cea60 con 0x7f6f0c1017d0 2024-03-28T18:35:11.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.207+0000 7f6f10ef7640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 0x7f6efc0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:11.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.211+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ef809bfa0 con 0x7f6f0c1017d0 2024-03-28T18:35:11.423 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.424+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f0c1a0440 con 0x7f6f0c1017d0 2024-03-28T18:35:11.424 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.424+0000 7f6f09ffb640 1 -- 172.21.15.145:0/1130480906 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f6ef809b8a0 con 0x7f6f0c1017d0 2024-03-28T18:35:11.424 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:11.424 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T18:35:11.424 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 1 2024-03-28T18:35:11.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 msgr2=0x7f6efc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 0x7f6efc0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f00005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 msgr2=0x7f6f0c197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:11.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800e9f0 tx=0x7f6ef800edf0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 shutdown_connections 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6efc0a2800 0x7f6efc0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f0c102190 0x7f6f0c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f0c1017d0 0x7f6f0c197f50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 --2- 172.21.15.145:0/1130480906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f0c1005e0 0x7f6f0c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 >> 172.21.15.145:0/1130480906 conn(0x7f6f0c0fbb50 msgr2=0x7f6f0c069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 shutdown_connections 2024-03-28T18:35:11.427 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:11.426+0000 7f6f1315b640 1 -- 172.21.15.145:0/1130480906 wait complete. 2024-03-28T18:35:11.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:11 smithi192 ceph-mon[36589]: Deploying daemon mds.l on smithi192 2024-03-28T18:35:11.692 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2969551417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:11.692 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2301889017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:11.692 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1130480906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:11.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:11 smithi145 ceph-mon[32151]: Deploying daemon mds.l on smithi192 2024-03-28T18:35:11.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2969551417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:11.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2301889017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:11.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1130480906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:11 smithi164 ceph-mon[36248]: Deploying daemon mds.l on smithi192 2024-03-28T18:35:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2969551417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2301889017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1130480906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:12.359 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 systemd[1]: Starting Ceph mds.l for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:12.631 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:12 smithi192 ceph-mon[36589]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:12.632 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 podman[65923]: 2024-03-28 18:35:12.262306638 +0000 UTC m=+0.017569266 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:12.632 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 podman[65923]: 2024-03-28 18:35:12.369641151 +0000 UTC m=+0.124903759 container create 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:35:12.632 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 podman[65923]: 2024-03-28 18:35:12.605896025 +0000 UTC m=+0.361158638 container init 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True) 2024-03-28T18:35:12.632 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 podman[65923]: 2024-03-28 18:35:12.608505789 +0000 UTC m=+0.363768411 container start 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T18:35:12.632 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 ceph-mds[65942]: -- 172.21.15.192:0/3651712038 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601c7629a40 con 0x5601c75fc800 2024-03-28T18:35:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:12 smithi145 ceph-mon[32151]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:12.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:12 smithi164 ceph-mon[36248]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:12.907 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: starting mds.l at 2024-03-28T18:35:12.907 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 bash[65923]: 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 2024-03-28T18:35:12.907 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:35:12 smithi192 systemd[1]: Started Ceph mds.l for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:13.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:13.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- 172.21.15.145:0/2311313169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4998103270 msgr2=0x7f4998109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:13.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2311313169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4998103270 0x7f4998109b00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4990009ca0 tx=0x7f499002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- 172.21.15.145:0/2311313169 shutdown_connections 2024-03-28T18:35:13.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2311313169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4998103270 0x7f4998109b00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2311313169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49981028b0 0x7f4998102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2311313169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- 172.21.15.145:0/2311313169 >> 172.21.15.145:0/2311313169 conn(0x7f49980fcc50 msgr2=0x7f49980ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- 172.21.15.145:0/2311313169 shutdown_connections 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- 172.21.15.145:0/2311313169 wait complete. 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 Processor -- start 2024-03-28T18:35:13.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.552+0000 7f499ea0b640 1 -- start start 2024-03-28T18:35:13.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:13.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:13.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4998103270 0x7f499819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:13.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499810d340 con 0x7f49981028b0 2024-03-28T18:35:13.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499810d1c0 con 0x7f4998103270 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499810d4c0 con 0x7f49981016c0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41104/0 (socket says 172.21.15.145:41104) 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f4997fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499cfa8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4998103270 0x7f499819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 -- 172.21.15.145:0/2576583229 learned_addr learned my addr 172.21.15.145:0/2576583229 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 -- 172.21.15.145:0/2576583229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 msgr2=0x7f4998197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 -- 172.21.15.145:0/2576583229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4998103270 msgr2=0x7f499819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4998103270 0x7f499819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 -- 172.21.15.145:0/2576583229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499819f580 con 0x7f49981028b0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f4997fff640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49977fe640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9f0 tx=0x7f498c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00cd80 con 0x7f49981028b0 2024-03-28T18:35:13.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.553+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499819f870 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.554+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f498c010910 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.554+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c0188f0 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.554+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499819fe10 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.554+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49981089e0 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.556+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f498c010ac0 con 0x7f49981028b0 2024-03-28T18:35:13.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.556+0000 7f49957fa640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 0x7f49800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:13.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.556+0000 7f4997fff640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 0x7f49800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:13.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.556+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f498c0cece0 con 0x7f49981028b0 2024-03-28T18:35:13.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.557+0000 7f4997fff640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 0x7f49800a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4998102710 tx=0x7f4988005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:13.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.560+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f498c09bbd0 con 0x7f49981028b0 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6832/3704007459,v1:172.21.15.192:6833/3704007459] up:boot 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: fsmap 1 up:standby 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:13.659 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:13.659 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:13 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6832/3704007459,v1:172.21.15.192:6833/3704007459] up:boot 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: fsmap 1 up:standby 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:13.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:13 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:13.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.777+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f499802d070 con 0x7f49981028b0 2024-03-28T18:35:13.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.777+0000 7f49957fa640 1 -- 172.21.15.145:0/2576583229 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f498c09fc70 con 0x7f49981028b0 2024-03-28T18:35:13.778 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:13.778 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:35:13.778 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 2 2024-03-28T18:35:13.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 msgr2=0x7f49800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:13.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 0x7f49800a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4998102710 tx=0x7f4988005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:13.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 msgr2=0x7f4998197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:13.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e9f0 tx=0x7f498c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 shutdown_connections 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4998103270 0x7f499819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49981028b0 0x7f4998197ea0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49800a2880 0x7f49800a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 --2- 172.21.15.145:0/2576583229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49981016c0 0x7f4998197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 >> 172.21.15.145:0/2576583229 conn(0x7f49980fcc50 msgr2=0x7f4998105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 shutdown_connections 2024-03-28T18:35:13.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:13.779+0000 7f499ea0b640 1 -- 172.21.15.145:0/2576583229 wait complete. 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6832/3704007459,v1:172.21.15.192:6833/3704007459] up:boot 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: fsmap 1 up:standby 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:13.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:13 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:14.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:14 smithi164 ceph-mon[36248]: Deploying daemon mds.k on smithi164 2024-03-28T18:35:14.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:14 smithi164 ceph-mon[36248]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:14.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:14 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2576583229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:14.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:14 smithi192 ceph-mon[36589]: Deploying daemon mds.k on smithi164 2024-03-28T18:35:14.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:14 smithi192 ceph-mon[36589]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:14.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:14 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2576583229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:14 smithi145 ceph-mon[32151]: Deploying daemon mds.k on smithi164 2024-03-28T18:35:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:14 smithi145 ceph-mon[32151]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:14 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2576583229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:15.174 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:14 smithi164 systemd[1]: Starting Ceph mds.k for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:15.435 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 podman[65847]: 2024-03-28 18:35:15.077529093 +0000 UTC m=+0.017611598 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:15.435 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 podman[65847]: 2024-03-28 18:35:15.191470473 +0000 UTC m=+0.131553003 container create 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:35:15.436 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 podman[65847]: 2024-03-28 18:35:15.436001659 +0000 UTC m=+0.376084150 container init 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T18:35:15.712 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 podman[65847]: 2024-03-28 18:35:15.438937167 +0000 UTC m=+0.379019666 container start 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:35:15.712 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 ceph-mds[65866]: -- 172.21.15.164:0/875722473 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e61d2c9a40 con 0x55e61d29d000 2024-03-28T18:35:15.712 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: starting mds.k at 2024-03-28T18:35:15.712 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 bash[65847]: 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 2024-03-28T18:35:15.712 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:35:15 smithi164 systemd[1]: Started Ceph mds.k for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:15.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 -- 172.21.15.145:0/3065434658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981032a0 msgr2=0x7f4e98109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/3065434658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981032a0 0x7f4e98109b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009ca0 tx=0x7f4e9002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 -- 172.21.15.145:0/3065434658 shutdown_connections 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/3065434658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981032a0 0x7f4e98109b30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/3065434658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981028e0 0x7f4e98102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/3065434658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 0x7f4e98101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 -- 172.21.15.145:0/3065434658 >> 172.21.15.145:0/3065434658 conn(0x7f4e980fcc40 msgr2=0x7f4e980ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 -- 172.21.15.145:0/3065434658 shutdown_connections 2024-03-28T18:35:15.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.894+0000 7f4e9dedc640 1 -- 172.21.15.145:0/3065434658 wait complete. 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 Processor -- start 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 -- start start 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 0x7f4e981979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981032a0 0x7f4e9819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810d370 con 0x7f4e981028e0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810d1f0 con 0x7f4e981016f0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e9dedc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9810d4f0 con 0x7f4e981032a0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e96ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e96ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41114/0 (socket says 172.21.15.145:41114) 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e96ffd640 1 -- 172.21.15.145:0/197206995 learned_addr learned my addr 172.21.15.145:0/197206995 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e977fe640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 0x7f4e981979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.895+0000 7f4e97fff640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981032a0 0x7f4e9819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 -- 172.21.15.145:0/197206995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981032a0 msgr2=0x7f4e9819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:15.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981032a0 0x7f4e9819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 -- 172.21.15.145:0/197206995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 msgr2=0x7f4e981979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 0x7f4e981979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 -- 172.21.15.145:0/197206995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9819f5d0 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e96ffd640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800d770 tx=0x7f4e8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88017070 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9819f8c0 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e88005500 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88004e00 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.896+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9819fdc0 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.897+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e981089b0 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.898+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4e88004fa0 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.899+0000 7f4e94ff9640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 0x7f4e800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.899+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4e880ce390 con 0x7f4e981028e0 2024-03-28T18:35:15.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.899+0000 7f4e977fe640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 0x7f4e800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:15.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.899+0000 7f4e977fe640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 0x7f4e800a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e98102740 tx=0x7f4e8c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:15.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:15.902+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e8809b280 con 0x7f4e981028e0 2024-03-28T18:35:16.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.149+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e9802d190 con 0x7f4e981028e0 2024-03-28T18:35:16.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.149+0000 7f4e94ff9640 1 -- 172.21.15.145:0/197206995 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f4e8809f320 con 0x7f4e981028e0 2024-03-28T18:35:16.155 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:16.155 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:35:16.155 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 3 2024-03-28T18:35:16.155 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 msgr2=0x7f4e800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:16.155 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 0x7f4e800a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e98102740 tx=0x7f4e8c00b390 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 msgr2=0x7f4e98197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800d770 tx=0x7f4e8800dc30 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 shutdown_connections 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4e800a2800 0x7f4e800a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e981032a0 0x7f4e9819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e981028e0 0x7f4e98197ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 --2- 172.21.15.145:0/197206995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e981016f0 0x7f4e981979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 >> 172.21.15.145:0/197206995 conn(0x7f4e980fcc40 msgr2=0x7f4e98105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 shutdown_connections 2024-03-28T18:35:16.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:16.152+0000 7f4e9dedc640 1 -- 172.21.15.145:0/197206995 wait complete. 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: Deploying daemon mds.j on smithi145 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6832/3835667312,v1:172.21.15.164:6833/3835667312] up:boot 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: fsmap 2 up:standby 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:35:16.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:16 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/197206995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:16.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: Deploying daemon mds.j on smithi145 2024-03-28T18:35:16.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6832/3835667312,v1:172.21.15.164:6833/3835667312] up:boot 2024-03-28T18:35:16.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: fsmap 2 up:standby 2024-03-28T18:35:16.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:35:16.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:16 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/197206995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: Deploying daemon mds.j on smithi145 2024-03-28T18:35:16.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6832/3835667312,v1:172.21.15.164:6833/3835667312] up:boot 2024-03-28T18:35:16.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: fsmap 2 up:standby 2024-03-28T18:35:16.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:35:16.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:16 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/197206995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:18.026 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:17 smithi145 systemd[1]: Starting Ceph mds.j for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:18.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 -- 172.21.15.145:0/459305293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 msgr2=0x7f64a0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/459305293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a0108a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6498009ca0 tx=0x7f649802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 -- 172.21.15.145:0/459305293 shutdown_connections 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/459305293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a0108a00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/459305293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 0x7f64a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/459305293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 0x7f64a01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.275+0000 7f64a4a0b640 1 -- 172.21.15.145:0/459305293 >> 172.21.15.145:0/459305293 conn(0x7f64a00fbb30 msgr2=0x7f64a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- 172.21.15.145:0/459305293 shutdown_connections 2024-03-28T18:35:18.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- 172.21.15.145:0/459305293 wait complete. 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 Processor -- start 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- start start 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 0x7f64a0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 0x7f64a019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a010c2b0 con 0x7f64a0102170 2024-03-28T18:35:18.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a010c130 con 0x7f64a01017b0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.276+0000 7f64a4a0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a010c430 con 0x7f64a01005c0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41128/0 (socket says 172.21.15.145:41128) 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 -- 172.21.15.145:0/793532160 learned_addr learned my addr 172.21.15.145:0/793532160 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f6497fff640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 0x7f64a019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649e59c640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 0x7f64a0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 -- 172.21.15.145:0/793532160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 msgr2=0x7f64a0069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 0x7f64a0069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 -- 172.21.15.145:0/793532160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 msgr2=0x7f64a019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 0x7f64a019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 -- 172.21.15.145:0/793532160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a019f5e0 con 0x7f64a0102170 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f649ed9d640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6498037040 tx=0x7f6498032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498039040 con 0x7f64a0102170 2024-03-28T18:35:18.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a019f810 con 0x7f64a0102170 2024-03-28T18:35:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6498044070 con 0x7f64a0102170 2024-03-28T18:35:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498004260 con 0x7f64a0102170 2024-03-28T18:35:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.277+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a019fd00 con 0x7f64a0102170 2024-03-28T18:35:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.278+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0107880 con 0x7f64a0102170 2024-03-28T18:35:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.279+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6498003970 con 0x7f64a0102170 2024-03-28T18:35:18.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.280+0000 7f64977fe640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 0x7f648c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:18.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.280+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64980f0d60 con 0x7f64a0102170 2024-03-28T18:35:18.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.280+0000 7f649e59c640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 0x7f648c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:18.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.280+0000 7f649e59c640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 0x7f648c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f64a0101610 tx=0x7f6490008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:18.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.284+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64980be2a0 con 0x7f64a0102170 2024-03-28T18:35:18.341 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 podman[69665]: 2024-03-28 18:35:17.975763068 +0000 UTC m=+0.017531075 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:18.341 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 podman[69665]: 2024-03-28 18:35:18.09697349 +0000 UTC m=+0.138741486 container create 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:18.341 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 podman[69665]: 2024-03-28 18:35:18.3419018 +0000 UTC m=+0.383669816 container init 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:18.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.517+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a002d270 con 0x7f64a0102170 2024-03-28T18:35:18.519 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.518+0000 7f64977fe640 1 -- 172.21.15.145:0/793532160 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f64980bdba0 con 0x7f64a0102170 2024-03-28T18:35:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:18.519 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:35:18.519 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 3 2024-03-28T18:35:18.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 msgr2=0x7f648c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:18.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 0x7f648c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f64a0101610 tx=0x7f6490008040 comp rx=0 tx=0).stop 2024-03-28T18:35:18.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 msgr2=0x7f64a019ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6498037040 tx=0x7f6498032f90 comp rx=0 tx=0).stop 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 shutdown_connections 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64a0102170 0x7f64a019ecd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64a01017b0 0x7f64a019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f648c0a2800 0x7f648c0a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 --2- 172.21.15.145:0/793532160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64a01005c0 0x7f64a0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 >> 172.21.15.145:0/793532160 conn(0x7f64a00fbb30 msgr2=0x7f64a0104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 shutdown_connections 2024-03-28T18:35:18.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:18.520+0000 7f64a4a0b640 1 -- 172.21.15.145:0/793532160 wait complete. 2024-03-28T18:35:18.594 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 podman[69665]: 2024-03-28 18:35:18.344806739 +0000 UTC m=+0.386574747 container start 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:18.594 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mds[69723]: -- 172.21.15.145:0/983750335 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5576a55cfa40 con 0x5576a55a2c00 2024-03-28T18:35:18.594 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j[69700]: starting mds.j at 2024-03-28T18:35:18.594 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 bash[69665]: 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c 2024-03-28T18:35:18.594 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:35:18 smithi145 systemd[1]: Started Ceph mds.j for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:18.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:18 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/793532160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:18.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:18 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/793532160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:19.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:19.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:19.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:19.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:18 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/793532160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:19.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:19 smithi145 ceph-mon[32151]: Deploying daemon mds.d on smithi145 2024-03-28T18:35:19.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:19 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6834/1607345926,v1:172.21.15.145:6835/1607345926] up:boot 2024-03-28T18:35:19.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:19 smithi145 ceph-mon[32151]: fsmap 3 up:standby 2024-03-28T18:35:19.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:35:19.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:19 smithi192 ceph-mon[36589]: Deploying daemon mds.d on smithi145 2024-03-28T18:35:19.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:19 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6834/1607345926,v1:172.21.15.145:6835/1607345926] up:boot 2024-03-28T18:35:19.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:19 smithi192 ceph-mon[36589]: fsmap 3 up:standby 2024-03-28T18:35:19.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:35:19.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:19 smithi164 ceph-mon[36248]: Deploying daemon mds.d on smithi145 2024-03-28T18:35:19.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:19 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6834/1607345926,v1:172.21.15.145:6835/1607345926] up:boot 2024-03-28T18:35:19.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:19 smithi164 ceph-mon[36248]: fsmap 3 up:standby 2024-03-28T18:35:19.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:35:20.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:20.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 -- 172.21.15.145:0/3414799861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94074080 msgr2=0x7fdd940744e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/3414799861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94074080 0x7fdd940744e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009900 tx=0x7fdd8802bcd0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 -- 172.21.15.145:0/3414799861 shutdown_connections 2024-03-28T18:35:20.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/3414799861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 0x7fdd9410ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/3414799861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94074080 0x7fdd940744e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/3414799861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94075660 0x7fdd94073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 -- 172.21.15.145:0/3414799861 >> 172.21.15.145:0/3414799861 conn(0x7fdd940fcc40 msgr2=0x7fdd940ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 -- 172.21.15.145:0/3414799861 shutdown_connections 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.648+0000 7fdd9a5df640 1 -- 172.21.15.145:0/3414799861 wait complete. 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 Processor -- start 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 -- start start 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94074080 0x7fdd94199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 0x7fdd941a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9410f550 con 0x7fdd94075660 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9410f3d0 con 0x7fdd94074080 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd9a5df640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9410f6d0 con 0x7fdd941095a0 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41166/0 (socket says 172.21.15.145:41166) 2024-03-28T18:35:20.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 -- 172.21.15.145:0/2534113153 learned_addr learned my addr 172.21.15.145:0/2534113153 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd93fff640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94074080 0x7fdd94199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd98b7c640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 0x7fdd941a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 -- 172.21.15.145:0/2534113153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 msgr2=0x7fdd941a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 0x7fdd941a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 -- 172.21.15.145:0/2534113153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94074080 msgr2=0x7fdd94199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.649+0000 7fdd937fe640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94074080 0x7fdd94199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd937fe640 1 -- 172.21.15.145:0/2534113153 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd941a17c0 con 0x7fdd94075660 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd937fe640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdd88005ab0 tx=0x7fdd88005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88041070 con 0x7fdd94075660 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd941a1a50 con 0x7fdd94075660 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd880054e0 con 0x7fdd94075660 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88034440 con 0x7fdd94075660 2024-03-28T18:35:20.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.650+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd941a1f90 con 0x7fdd94075660 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.651+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd94108360 con 0x7fdd94075660 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.652+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdd88044070 con 0x7fdd94075660 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.652+0000 7fdd917fa640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 0x7fdd7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.653+0000 7fdd93fff640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 0x7fdd7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.653+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdd880f05a0 con 0x7fdd94075660 2024-03-28T18:35:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.653+0000 7fdd93fff640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 0x7fdd7c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400bbe0 tx=0x7fdd8400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:20.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.657+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd880bdae0 con 0x7fdd94075660 2024-03-28T18:35:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.899+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9402d240 con 0x7fdd94075660 2024-03-28T18:35:20.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.900+0000 7fdd917fa640 1 -- 172.21.15.145:0/2534113153 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2916 (secure 0 0 0) 0x7fdd880bd3e0 con 0x7fdd94075660 2024-03-28T18:35:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:20.901 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:35:20.901 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 4 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 msgr2=0x7fdd7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 0x7fdd7c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400bbe0 tx=0x7fdd8400b390 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 msgr2=0x7fdd9419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdd88005ab0 tx=0x7fdd88005de0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 shutdown_connections 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdd7c0a2800 0x7fdd7c0a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd941095a0 0x7fdd941a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd94075660 0x7fdd9419a0e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 --2- 172.21.15.145:0/2534113153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd94074080 0x7fdd94199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 >> 172.21.15.145:0/2534113153 conn(0x7fdd940fcc40 msgr2=0x7fdd94105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:20.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 shutdown_connections 2024-03-28T18:35:20.904 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:20.902+0000 7fdd9a5df640 1 -- 172.21.15.145:0/2534113153 wait complete. 2024-03-28T18:35:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:20 smithi192 ceph-mon[36589]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:20.913 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:20 smithi145 ceph-mon[32151]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:20.913 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:20 smithi145 systemd[1]: Starting Ceph mds.d for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:20.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:20 smithi164 ceph-mon[36248]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:21.265 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:20 smithi145 podman[70048]: 2024-03-28 18:35:20.892626543 +0000 UTC m=+0.022055103 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:21.265 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 podman[70048]: 2024-03-28 18:35:21.021170671 +0000 UTC m=+0.150599272 container create 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:21.265 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 podman[70048]: 2024-03-28 18:35:21.265812037 +0000 UTC m=+0.395240599 container init 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:35:21.526 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 podman[70048]: 2024-03-28 18:35:21.26869607 +0000 UTC m=+0.398124632 container start 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:35:21.526 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mds[70069]: -- 172.21.15.145:0/3536629134 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c4549d1a40 con 0x55c4549a4c00 2024-03-28T18:35:21.527 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: starting mds.d at 2024-03-28T18:35:21.527 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 bash[70048]: 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 2024-03-28T18:35:21.527 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:35:21 smithi145 systemd[1]: Started Ceph mds.d for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2534113153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:21.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:21 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2534113153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:21 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2534113153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:22.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:21 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:22 smithi164 ceph-mon[36248]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:22 smithi164 ceph-mon[36248]: Deploying daemon mds.e on smithi164 2024-03-28T18:35:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:22 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6836/3754178140,v1:172.21.15.145:6837/3754178140] up:boot 2024-03-28T18:35:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:22 smithi164 ceph-mon[36248]: fsmap 4 up:standby 2024-03-28T18:35:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:22 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:35:22.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:22 smithi192 ceph-mon[36589]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:22.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:22 smithi192 ceph-mon[36589]: Deploying daemon mds.e on smithi164 2024-03-28T18:35:22.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:22 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6836/3754178140,v1:172.21.15.145:6837/3754178140] up:boot 2024-03-28T18:35:22.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:22 smithi192 ceph-mon[36589]: fsmap 4 up:standby 2024-03-28T18:35:22.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:22 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:35:22.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:22.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:22 smithi145 ceph-mon[32151]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:22.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:22 smithi145 ceph-mon[32151]: Deploying daemon mds.e on smithi164 2024-03-28T18:35:22.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:22 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6836/3754178140,v1:172.21.15.145:6837/3754178140] up:boot 2024-03-28T18:35:22.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:22 smithi145 ceph-mon[32151]: fsmap 4 up:standby 2024-03-28T18:35:22.945 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:22 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:35:23.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 -- 172.21.15.145:0/317849584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0881017c0 msgr2=0x7fd088101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:23.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 --2- 172.21.15.145:0/317849584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0881017c0 0x7fd088101c40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd070009900 tx=0x7fd07002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:23.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 -- 172.21.15.145:0/317849584 shutdown_connections 2024-03-28T18:35:23.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 --2- 172.21.15.145:0/317849584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd088102180 0x7fd088108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 --2- 172.21.15.145:0/317849584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0881017c0 0x7fd088101c40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 --2- 172.21.15.145:0/317849584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 0x7fd0881009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 -- 172.21.15.145:0/317849584 >> 172.21.15.145:0/317849584 conn(0x7fd0880fbb80 msgr2=0x7fd0880fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 -- 172.21.15.145:0/317849584 shutdown_connections 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.032+0000 7fd08d480640 1 -- 172.21.15.145:0/317849584 wait complete. 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 Processor -- start 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 -- start start 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 0x7fd088199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0881017c0 0x7fd08819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08810c230 con 0x7fd088102180 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08810c0b0 con 0x7fd0881017c0 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd08d480640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08810c3b0 con 0x7fd0881005d0 2024-03-28T18:35:23.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37342/0 (socket says 172.21.15.145:37342) 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 -- 172.21.15.145:0/3901261891 learned_addr learned my addr 172.21.15.145:0/3901261891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd07ffff640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0881017c0 0x7fd08819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd086ffd640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 0x7fd088199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 -- 172.21.15.145:0/3901261891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 msgr2=0x7fd088199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 0x7fd088199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 -- 172.21.15.145:0/3901261891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0881017c0 msgr2=0x7fd08819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0881017c0 0x7fd08819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.033+0000 7fd0877fe640 1 -- 172.21.15.145:0/3901261891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0881a1810 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd0877fe640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd08000eaa0 tx=0x7fd08000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08000cdb0 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd080005020 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0881a1b00 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800116b0 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.034+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0881a1fd0 con 0x7fd088102180 2024-03-28T18:35:23.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.035+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0881a22c0 con 0x7fd088102180 2024-03-28T18:35:23.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.036+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd080005560 con 0x7fd088102180 2024-03-28T18:35:23.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.037+0000 7fd084ff9640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 0x7fd0740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:23.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.037+0000 7fd086ffd640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 0x7fd0740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:23.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.037+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd0800cec30 con 0x7fd088102180 2024-03-28T18:35:23.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.037+0000 7fd086ffd640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 0x7fd0740a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd078005fd0 tx=0x7fd078005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:23.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.041+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd080016030 con 0x7fd088102180 2024-03-28T18:35:23.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.253+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0881a2530 con 0x7fd088102180 2024-03-28T18:35:23.254 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.254+0000 7fd084ff9640 1 -- 172.21.15.145:0/3901261891 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7fd08009c170 con 0x7fd088102180 2024-03-28T18:35:23.254 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:23.254 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:35:23.254 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 5 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 msgr2=0x7fd0740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 0x7fd0740a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd078005fd0 tx=0x7fd078005850 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 msgr2=0x7fd0881a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd08000eaa0 tx=0x7fd08000ef60 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 shutdown_connections 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd088102180 0x7fd0881a11a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0881017c0 0x7fd08819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd0740a2800 0x7fd0740a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 --2- 172.21.15.145:0/3901261891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0881005d0 0x7fd088199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:23.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 >> 172.21.15.145:0/3901261891 conn(0x7fd0880fbb80 msgr2=0x7fd0880fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:23.257 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 shutdown_connections 2024-03-28T18:35:23.257 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:23.256+0000 7fd08d480640 1 -- 172.21.15.145:0/3901261891 wait complete. 2024-03-28T18:35:23.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:23 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3901261891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:23.712 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:23 smithi164 systemd[1]: Starting Ceph mds.e for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:23.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:23 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3901261891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:24.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:23 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3901261891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:24.140 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:23 smithi164 podman[66166]: 2024-03-28 18:35:23.773422296 +0000 UTC m=+0.017901096 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:24.141 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:23 smithi164 podman[66166]: 2024-03-28 18:35:23.880058574 +0000 UTC m=+0.124537362 container create 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:35:24.141 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 podman[66166]: 2024-03-28 18:35:24.141548712 +0000 UTC m=+0.386027506 container init 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:35:24.462 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 podman[66166]: 2024-03-28 18:35:24.144196861 +0000 UTC m=+0.388675650 container start 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T18:35:24.462 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mds[66185]: -- 172.21.15.164:0/3656216997 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f5ef0ba40 con 0x564f5eedec00 2024-03-28T18:35:24.462 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: starting mds.e at 2024-03-28T18:35:24.462 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 bash[66166]: 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 2024-03-28T18:35:24.462 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:35:24 smithi164 systemd[1]: Started Ceph mds.e for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:25.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:25.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:25.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:25.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:25.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:25.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:25.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:25.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 -- 172.21.15.145:0/2298293821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 msgr2=0x7fc624100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 --2- 172.21.15.145:0/2298293821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc624100a00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc618009900 tx=0x7fc61802bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 -- 172.21.15.145:0/2298293821 shutdown_connections 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 --2- 172.21.15.145:0/2298293821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc624102190 0x7fc624108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 --2- 172.21.15.145:0/2298293821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6241017d0 0x7fc624101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 --2- 172.21.15.145:0/2298293821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc624100a00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 -- 172.21.15.145:0/2298293821 >> 172.21.15.145:0/2298293821 conn(0x7fc6240fbb50 msgr2=0x7fc6240fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:25.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 -- 172.21.15.145:0/2298293821 shutdown_connections 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.383+0000 7fc62884b640 1 -- 172.21.15.145:0/2298293821 wait complete. 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 Processor -- start 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 -- start start 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 0x7fc62419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc624102190 0x7fc62419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62410c2b0 con 0x7fc6241005e0 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62410c130 con 0x7fc6241017d0 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62884b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62410c430 con 0x7fc624102190 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37356/0 (socket says 172.21.15.145:37356) 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 -- 172.21.15.145:0/3831368516 learned_addr learned my addr 172.21.15.145:0/3831368516 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:25.386 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc621d9b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 0x7fc62419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 -- 172.21.15.145:0/3831368516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc624102190 msgr2=0x7fc62419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc624102190 0x7fc62419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 -- 172.21.15.145:0/3831368516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 msgr2=0x7fc62419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 0x7fc62419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 -- 172.21.15.145:0/3831368516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62419f640 con 0x7fc6241005e0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc621d9b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 0x7fc62419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.384+0000 7fc62259c640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc618000c00 tx=0x7fc61802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.385+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618043020 con 0x7fc6241005e0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.385+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6180054e0 con 0x7fc6241005e0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.385+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62419f8d0 con 0x7fc6241005e0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.385+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618003ce0 con 0x7fc6241005e0 2024-03-28T18:35:25.387 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.385+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62419fe40 con 0x7fc6241005e0 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.386+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624107900 con 0x7fc6241005e0 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.387+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc618046020 con 0x7fc6241005e0 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.387+0000 7fc6137fe640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 0x7fc60c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.387+0000 7fc621d9b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 0x7fc60c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.387+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc6180f07d0 con 0x7fc6241005e0 2024-03-28T18:35:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.388+0000 7fc621d9b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 0x7fc60c0a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc62406a610 tx=0x7fc614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:25.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.391+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc61804d020 con 0x7fc6241005e0 2024-03-28T18:35:25.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.604+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6240622a0 con 0x7fc6241005e0 2024-03-28T18:35:25.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.605+0000 7fc6137fe640 1 -- 172.21.15.145:0/3831368516 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4504 (secure 0 0 0) 0x7fc6180bdd10 con 0x7fc6241005e0 2024-03-28T18:35:25.605 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:25.605 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T18:35:25.605 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 6 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 msgr2=0x7fc60c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 0x7fc60c0a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc62406a610 tx=0x7fc614005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 msgr2=0x7fc62406a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc618000c00 tx=0x7fc61802c6c0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 shutdown_connections 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc60c0a2880 0x7fc60c0a4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc624102190 0x7fc62419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6241017d0 0x7fc62419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 --2- 172.21.15.145:0/3831368516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6241005e0 0x7fc62406a8a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 >> 172.21.15.145:0/3831368516 conn(0x7fc6240fbb50 msgr2=0x7fc624104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:25.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 shutdown_connections 2024-03-28T18:35:25.608 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:25.607+0000 7fc62884b640 1 -- 172.21.15.145:0/3831368516 wait complete. 2024-03-28T18:35:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:25 smithi192 ceph-mon[36589]: Deploying daemon mds.f on smithi192 2024-03-28T18:35:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:25 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6834/3793155920,v1:172.21.15.164:6835/3793155920] up:boot 2024-03-28T18:35:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:25 smithi192 ceph-mon[36589]: fsmap 5 up:standby 2024-03-28T18:35:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:25 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:35:25.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:25 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3831368516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:25 smithi164 ceph-mon[36248]: Deploying daemon mds.f on smithi192 2024-03-28T18:35:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:25 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6834/3793155920,v1:172.21.15.164:6835/3793155920] up:boot 2024-03-28T18:35:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:25 smithi164 ceph-mon[36248]: fsmap 5 up:standby 2024-03-28T18:35:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:25 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:35:25.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:25 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3831368516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:25 smithi145 ceph-mon[32151]: Deploying daemon mds.f on smithi192 2024-03-28T18:35:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:25 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6834/3793155920,v1:172.21.15.164:6835/3793155920] up:boot 2024-03-28T18:35:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:25 smithi145 ceph-mon[32151]: fsmap 5 up:standby 2024-03-28T18:35:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:25 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:35:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:25 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3831368516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:26.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:26 smithi192 ceph-mon[36589]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:26.861 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:26 smithi192 systemd[1]: Starting Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:26 smithi164 ceph-mon[36248]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:27.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:26 smithi145 ceph-mon[32151]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:27.135 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:26 smithi192 podman[66235]: 2024-03-28 18:35:26.761663157 +0000 UTC m=+0.017877968 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:27.136 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:26 smithi192 podman[66235]: 2024-03-28 18:35:26.878382557 +0000 UTC m=+0.134597378 container create 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:35:27.136 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 podman[66235]: 2024-03-28 18:35:27.136366972 +0000 UTC m=+0.392581773 container init 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T18:35:27.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 podman[66235]: 2024-03-28 18:35:27.138950654 +0000 UTC m=+0.395165457 container start 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:35:27.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 ceph-mds[66254]: -- 172.21.15.192:0/1912066107 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a99271a40 con 0x557a99244800 2024-03-28T18:35:27.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: starting mds.f at 2024-03-28T18:35:27.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 bash[66235]: 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 2024-03-28T18:35:27.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:35:27 smithi192 systemd[1]: Started Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:27.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 -- 172.21.15.145:0/615920977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f30100910 msgr2=0x7f6f30100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 --2- 172.21.15.145:0/615920977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f30100910 0x7f6f30100d70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f24009900 tx=0x7f6f2402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 -- 172.21.15.145:0/615920977 shutdown_connections 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 --2- 172.21.15.145:0/615920977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f301012b0 0x7f6f30101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 --2- 172.21.15.145:0/615920977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f30100910 0x7f6f30100d70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 --2- 172.21.15.145:0/615920977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f300fffb0 0x7f6f301003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 -- 172.21.15.145:0/615920977 >> 172.21.15.145:0/615920977 conn(0x7f6f300fbb30 msgr2=0x7f6f300fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:27.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 -- 172.21.15.145:0/615920977 shutdown_connections 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.732+0000 7f6f3595b640 1 -- 172.21.15.145:0/615920977 wait complete. 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 Processor -- start 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 -- start start 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f30100910 0x7f6f3019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f301012b0 0x7f6f301a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3010f540 con 0x7f6f300fffb0 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3010f3c0 con 0x7f6f30100910 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f3595b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3010f6c0 con 0x7f6f301012b0 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f2effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f2effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37382/0 (socket says 172.21.15.145:37382) 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f2effd640 1 -- 172.21.15.145:0/1014292510 learned_addr learned my addr 172.21.15.145:0/1014292510 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:27.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.733+0000 7f6f2f7fe640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f301012b0 0x7f6f301a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 -- 172.21.15.145:0/1014292510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f301012b0 msgr2=0x7f6f301a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2e7fc640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f30100910 0x7f6f3019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f301012b0 0x7f6f301a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 -- 172.21.15.145:0/1014292510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f30100910 msgr2=0x7f6f3019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f30100910 0x7f6f3019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 -- 172.21.15.145:0/1014292510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f301a1870 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f2effd640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000eb90 tx=0x7f6f2000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004d20 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f301a1b60 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f2000ee90 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20010830 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.734+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f301a2060 con 0x7f6f300fffb0 2024-03-28T18:35:27.737 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.735+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3010aed0 con 0x7f6f300fffb0 2024-03-28T18:35:27.738 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.736+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6f20005020 con 0x7f6f300fffb0 2024-03-28T18:35:27.738 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.737+0000 7f6f34959640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:27.738 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.737+0000 7f6f2e7fc640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:27.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.737+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f200ced30 con 0x7f6f300fffb0 2024-03-28T18:35:27.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.737+0000 7f6f2e7fc640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f24036040 tx=0x7f6f24000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:27.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.741+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f2009c270 con 0x7f6f300fffb0 2024-03-28T18:35:27.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.956+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f3002d070 con 0x7f6f300fffb0 2024-03-28T18:35:27.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.957+0000 7f6f34959640 1 -- 172.21.15.145:0/1014292510 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5298 (secure 0 0 0) 0x7f6f2009bb70 con 0x7f6f300fffb0 2024-03-28T18:35:27.957 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:27.957 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T18:35:27.958 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 7 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 msgr2=0x7f6f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6f24036040 tx=0x7f6f24000bc0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 msgr2=0x7f6f30199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000eb90 tx=0x7f6f2000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 shutdown_connections 2024-03-28T18:35:27.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f301012b0 0x7f6f301a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f30100910 0x7f6f3019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 --2- 172.21.15.145:0/1014292510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f300fffb0 0x7f6f30199c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 >> 172.21.15.145:0/1014292510 conn(0x7f6f300fbb30 msgr2=0x7f6f300fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 shutdown_connections 2024-03-28T18:35:27.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:27.959+0000 7f6f3595b640 1 -- 172.21.15.145:0/1014292510 wait complete. 2024-03-28T18:35:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: Deploying daemon mds.c on smithi192 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6834/1226474966,v1:172.21.15.192:6835/1226474966] up:boot 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: fsmap 6 up:standby 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:35:28.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:28 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1014292510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: Deploying daemon mds.c on smithi192 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6834/1226474966,v1:172.21.15.192:6835/1226474966] up:boot 2024-03-28T18:35:28.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: fsmap 6 up:standby 2024-03-28T18:35:28.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:35:28.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:28 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1014292510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:28.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: Deploying daemon mds.c on smithi192 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6834/1226474966,v1:172.21.15.192:6835/1226474966] up:boot 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: fsmap 6 up:standby 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:35:28.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:28 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1014292510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:29.407 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 systemd[1]: Starting Ceph mds.c for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:29.758 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 podman[66544]: 2024-03-28 18:35:29.393805793 +0000 UTC m=+0.017985824 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:29.758 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 podman[66544]: 2024-03-28 18:35:29.496775958 +0000 UTC m=+0.120955995 container create 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T18:35:29.758 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 podman[66544]: 2024-03-28 18:35:29.73292894 +0000 UTC m=+0.357108958 container init 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:35:29.758 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 podman[66544]: 2024-03-28 18:35:29.735619193 +0000 UTC m=+0.359799202 container start 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:35:29.758 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 ceph-mds[66563]: -- 172.21.15.192:0/950260457 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ad30cb1a40 con 0x55ad30c84800 2024-03-28T18:35:29.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:30.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 -- 172.21.15.145:0/3950874039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c104af0 msgr2=0x7f648c106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 --2- 172.21.15.145:0/3950874039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c104af0 0x7f648c106ee0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6478009900 tx=0x7f647802bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 -- 172.21.15.145:0/3950874039 shutdown_connections 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 --2- 172.21.15.145:0/3950874039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 0x7f648c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 --2- 172.21.15.145:0/3950874039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c104af0 0x7f648c106ee0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 --2- 172.21.15.145:0/3950874039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c1021c0 0x7f648c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 -- 172.21.15.145:0/3950874039 >> 172.21.15.145:0/3950874039 conn(0x7f648c0fbbd0 msgr2=0x7f648c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 -- 172.21.15.145:0/3950874039 shutdown_connections 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.082+0000 7f6493036640 1 -- 172.21.15.145:0/3950874039 wait complete. 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 Processor -- start 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 -- start start 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c104af0 0x7f648c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 0x7f648c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c10d340 con 0x7f648c1021c0 2024-03-28T18:35:30.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c10d1c0 con 0x7f648c107420 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6493036640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c10d4c0 con 0x7f648c104af0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37400/0 (socket says 172.21.15.145:37400) 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 -- 172.21.15.145:0/2426614336 learned_addr learned my addr 172.21.15.145:0/2426614336 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f64915d3640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 0x7f648c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f648bfff640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c104af0 0x7f648c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 -- 172.21.15.145:0/2426614336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c104af0 msgr2=0x7f648c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c104af0 0x7f648c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 -- 172.21.15.145:0/2426614336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 msgr2=0x7f648c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 0x7f648c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.083+0000 7f6490dd2640 1 -- 172.21.15.145:0/2426614336 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c19f640 con 0x7f648c1021c0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6490dd2640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f648000d950 tx=0x7f648000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:30.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480017070 con 0x7f648c1021c0 2024-03-28T18:35:30.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6480009c30 con 0x7f648c1021c0 2024-03-28T18:35:30.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c19f930 con 0x7f648c1021c0 2024-03-28T18:35:30.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648000f6b0 con 0x7f648c1021c0 2024-03-28T18:35:30.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.084+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c19fe50 con 0x7f648c1021c0 2024-03-28T18:35:30.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.085+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c0fd6c0 con 0x7f648c1021c0 2024-03-28T18:35:30.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.086+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f64800109d0 con 0x7f648c1021c0 2024-03-28T18:35:30.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.087+0000 7f6489ffb640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:30.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.087+0000 7f648bfff640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:30.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.087+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64800ce1c0 con 0x7f648c1021c0 2024-03-28T18:35:30.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.087+0000 7f648bfff640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 0x7f647c0a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6478000c00 tx=0x7f6478005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:30.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.091+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6480009de0 con 0x7f648c1021c0 2024-03-28T18:35:30.157 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: starting mds.c at 2024-03-28T18:35:30.157 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 bash[66544]: 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 2024-03-28T18:35:30.157 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:35:29 smithi192 systemd[1]: Started Ceph mds.c for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:30.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.304+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648c02d070 con 0x7f648c1021c0 2024-03-28T18:35:30.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.305+0000 7f6489ffb640 1 -- 172.21.15.145:0/2426614336 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5298 (secure 0 0 0) 0x7f648c02d070 con 0x7f648c1021c0 2024-03-28T18:35:30.305 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:30.305 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T18:35:30.305 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 7 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 msgr2=0x7f647c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 0x7f647c0a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6478000c00 tx=0x7f6478005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 msgr2=0x7f648c197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f648000d950 tx=0x7f648000de10 comp rx=0 tx=0).stop 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 shutdown_connections 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c107420 0x7f648c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f648c104af0 0x7f648c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 --2- 172.21.15.145:0/2426614336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f648c1021c0 0x7f648c197a20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:30.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 >> 172.21.15.145:0/2426614336 conn(0x7f648c0fbbd0 msgr2=0x7f648c102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:30.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 shutdown_connections 2024-03-28T18:35:30.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:30.307+0000 7f6493036640 1 -- 172.21.15.145:0/2426614336 wait complete. 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:30.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:30.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:30 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2426614336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:30.819 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:30.820 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:30 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2426614336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:30.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:30 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2426614336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:31.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:31 smithi164 ceph-mon[36248]: Deploying daemon mds.h on smithi164 2024-03-28T18:35:31.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:31 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:boot 2024-03-28T18:35:31.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:31 smithi164 ceph-mon[36248]: fsmap 7 up:standby 2024-03-28T18:35:31.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:31 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:35:31.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:31 smithi192 ceph-mon[36589]: Deploying daemon mds.h on smithi164 2024-03-28T18:35:31.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:31 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:boot 2024-03-28T18:35:31.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:31 smithi192 ceph-mon[36589]: fsmap 7 up:standby 2024-03-28T18:35:31.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:31 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:35:32.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:31 smithi145 ceph-mon[32151]: Deploying daemon mds.h on smithi164 2024-03-28T18:35:32.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:31 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:boot 2024-03-28T18:35:32.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:31 smithi145 ceph-mon[32151]: fsmap 7 up:standby 2024-03-28T18:35:32.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:35:32.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 -- 172.21.15.145:0/2734425508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c1017b0 msgr2=0x7f612c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 --2- 172.21.15.145:0/2734425508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c1017b0 0x7f612c101c30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6120009900 tx=0x7f612002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 -- 172.21.15.145:0/2734425508 shutdown_connections 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 --2- 172.21.15.145:0/2734425508 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c102170 0x7f612c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 --2- 172.21.15.145:0/2734425508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c1017b0 0x7f612c101c30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 --2- 172.21.15.145:0/2734425508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1005c0 0x7f612c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 -- 172.21.15.145:0/2734425508 >> 172.21.15.145:0/2734425508 conn(0x7f612c0fbb30 msgr2=0x7f612c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 -- 172.21.15.145:0/2734425508 shutdown_connections 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.434+0000 7f6130c05640 1 -- 172.21.15.145:0/2734425508 wait complete. 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 Processor -- start 2024-03-28T18:35:32.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 -- start start 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 0x7f612c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1017b0 0x7f612c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c10c240 con 0x7f612c102170 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c10c0c0 con 0x7f612c1005c0 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6130c05640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c10c3c0 con 0x7f612c1017b0 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37418/0 (socket says 172.21.15.145:37418) 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 0x7f612c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 -- 172.21.15.145:0/3316551269 learned_addr learned my addr 172.21.15.145:0/3316551269 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f6129d9b640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1017b0 0x7f612c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 -- 172.21.15.145:0/3316551269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1017b0 msgr2=0x7f612c19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1017b0 0x7f612c19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 -- 172.21.15.145:0/3316551269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 msgr2=0x7f612c069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 0x7f612c069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 -- 172.21.15.145:0/3316551269 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c19f610 con 0x7f612c102170 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612a59c640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 0x7f612c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f612ad9d640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f612400d950 tx=0x7f612400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.435+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124017030 con 0x7f612c102170 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.436+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c19f900 con 0x7f612c102170 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.436+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6124004020 con 0x7f612c102170 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.436+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124003c00 con 0x7f612c102170 2024-03-28T18:35:32.438 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.436+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c19fdd0 con 0x7f612c102170 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.437+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c107960 con 0x7f612c102170 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.438+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f612401a070 con 0x7f612c102170 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.438+0000 7f611b7fe640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 0x7f61140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.438+0000 7f612a59c640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 0x7f61140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.438+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f61240cde10 con 0x7f612c102170 2024-03-28T18:35:32.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.439+0000 7f612a59c640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 0x7f61140a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f611c009fd0 tx=0x7f611c009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:32.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.443+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6124021070 con 0x7f612c102170 2024-03-28T18:35:32.534 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 podman[66478]: 2024-03-28 18:35:32.46151224 +0000 UTC m=+0.140808912 container create d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-28T18:35:32.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.657+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f612c02d070 con 0x7f612c102170 2024-03-28T18:35:32.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.657+0000 7f611b7fe640 1 -- 172.21.15.145:0/3316551269 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6092 (secure 0 0 0) 0x7f612409b350 con 0x7f612c102170 2024-03-28T18:35:32.657 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:32.658 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:35:32.658 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 8 2024-03-28T18:35:32.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 msgr2=0x7f61140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 0x7f61140a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f611c009fd0 tx=0x7f611c009680 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 msgr2=0x7f612c19ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f612400d950 tx=0x7f612400de10 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 shutdown_connections 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f612c102170 0x7f612c19ed50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f61140a2800 0x7f61140a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f612c1017b0 0x7f612c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 --2- 172.21.15.145:0/3316551269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f612c1005c0 0x7f612c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.659+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 >> 172.21.15.145:0/3316551269 conn(0x7f612c0fbb30 msgr2=0x7f612c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:32.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.660+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 shutdown_connections 2024-03-28T18:35:32.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:32.660+0000 7f6130c05640 1 -- 172.21.15.145:0/3316551269 wait complete. 2024-03-28T18:35:32.669 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:32 smithi145 ceph-mon[32151]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:32.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:32 smithi192 ceph-mon[36589]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:32.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:32 smithi164 ceph-mon[36248]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:32.831 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 podman[66478]: 2024-03-28 18:35:32.714312258 +0000 UTC m=+0.393608881 container init d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:32.831 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 podman[66478]: 2024-03-28 18:35:32.717151534 +0000 UTC m=+0.396448152 container start d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T18:35:32.831 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 ceph-mds[66497]: -- 172.21.15.164:0/620493428 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c1beac7a40 con 0x55c1bea9b000 2024-03-28T18:35:32.831 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: starting mds.h at 2024-03-28T18:35:33.212 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 bash[66478]: d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 2024-03-28T18:35:33.212 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:35:32 smithi164 systemd[1]: Started Ceph mds.h for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3316551269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:33.803 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:33 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3316551269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:33.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:33.874 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:33 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3316551269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:33 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:34.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:34.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- 172.21.15.145:0/2791421729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 msgr2=0x7f90c4104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:34.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 --2- 172.21.15.145:0/2791421729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4104590 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009900 tx=0x7f90b402bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- 172.21.15.145:0/2791421729 shutdown_connections 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 --2- 172.21.15.145:0/2791421729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4107400 0x7f90c41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 --2- 172.21.15.145:0/2791421729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4104ad0 0x7f90c4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 --2- 172.21.15.145:0/2791421729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4104590 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- 172.21.15.145:0/2791421729 >> 172.21.15.145:0/2791421729 conn(0x7f90c40fbb30 msgr2=0x7f90c40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- 172.21.15.145:0/2791421729 shutdown_connections 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- 172.21.15.145:0/2791421729 wait complete. 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 Processor -- start 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 -- start start 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.788+0000 7f90c9663640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4104ad0 0x7f90c4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 0x7f90c419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:34.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c410d380 con 0x7f90c41021a0 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c410d200 con 0x7f90c4107400 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c410d500 con 0x7f90c4104ad0 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54390/0 (socket says 172.21.15.145:54390) 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 -- 172.21.15.145:0/2984207271 learned_addr learned my addr 172.21.15.145:0/2984207271 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c27fc640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4104ad0 0x7f90c4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c37fe640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 0x7f90c419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 -- 172.21.15.145:0/2984207271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4104ad0 msgr2=0x7f90c4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4104ad0 0x7f90c4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 -- 172.21.15.145:0/2984207271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 msgr2=0x7f90c419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 0x7f90c419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:34.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 -- 172.21.15.145:0/2984207271 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c419f580 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c37fe640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 0x7f90c419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c2ffd640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000c00 tx=0x7f90b4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4043020 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c419f810 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90b40054e0 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4003df0 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.789+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c419fd60 con 0x7f90c41021a0 2024-03-28T18:35:34.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.790+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c40fd620 con 0x7f90c41021a0 2024-03-28T18:35:34.794 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.791+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f90b4046020 con 0x7f90c41021a0 2024-03-28T18:35:34.794 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.792+0000 7f90abfff640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 0x7f90ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:34.794 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.792+0000 7f90c27fc640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 0x7f90ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:34.794 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.792+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90b40f0c50 con 0x7f90c41021a0 2024-03-28T18:35:34.794 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.792+0000 7f90c27fc640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 0x7f90ac0a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f90c4198e70 tx=0x7f90b800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:34.795 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:34.795+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90b404d020 con 0x7f90c41021a0 2024-03-28T18:35:34.833 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:34 smithi192 ceph-mon[36589]: Deploying daemon mds.i on smithi192 2024-03-28T18:35:34.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:34 smithi192 ceph-mon[36589]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:34.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:34 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:boot 2024-03-28T18:35:34.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:34 smithi192 ceph-mon[36589]: fsmap 8 up:standby 2024-03-28T18:35:34.834 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:34 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:35:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:34 smithi164 ceph-mon[36248]: Deploying daemon mds.i on smithi192 2024-03-28T18:35:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:34 smithi164 ceph-mon[36248]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:34 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:boot 2024-03-28T18:35:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:34 smithi164 ceph-mon[36248]: fsmap 8 up:standby 2024-03-28T18:35:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:34 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:35:35.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.007+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90c402d070 con 0x7f90c41021a0 2024-03-28T18:35:35.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.008+0000 7f90abfff640 1 -- 172.21.15.145:0/2984207271 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6886 (secure 0 0 0) 0x7f90b40be190 con 0x7f90c41021a0 2024-03-28T18:35:35.008 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:35.008 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:35:35.008 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 9 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 msgr2=0x7f90ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 0x7f90ac0a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f90c4198e70 tx=0x7f90b800a2d0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 msgr2=0x7f90c4197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000c00 tx=0x7f90b4005dd0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 shutdown_connections 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90c4107400 0x7f90c419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90ac0a2800 0x7f90ac0a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90c4104ad0 0x7f90c4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 --2- 172.21.15.145:0/2984207271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90c41021a0 0x7f90c4197960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 >> 172.21.15.145:0/2984207271 conn(0x7f90c40fbb30 msgr2=0x7f90c4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 shutdown_connections 2024-03-28T18:35:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:35.010+0000 7f90c9663640 1 -- 172.21.15.145:0/2984207271 wait complete. 2024-03-28T18:35:35.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:34 smithi145 ceph-mon[32151]: Deploying daemon mds.i on smithi192 2024-03-28T18:35:35.019 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:34 smithi145 ceph-mon[32151]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:35.020 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:34 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:boot 2024-03-28T18:35:35.020 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:34 smithi145 ceph-mon[32151]: fsmap 8 up:standby 2024-03-28T18:35:35.020 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:34 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:35:35.157 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 systemd[1]: Starting Ceph mds.i for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:35.487 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 podman[66860]: 2024-03-28 18:35:35.140152781 +0000 UTC m=+0.017782157 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:35.488 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 podman[66860]: 2024-03-28 18:35:35.252350476 +0000 UTC m=+0.129979832 container create 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T18:35:35.488 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 podman[66860]: 2024-03-28 18:35:35.488496053 +0000 UTC m=+0.366125409 container init 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2984207271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 podman[66860]: 2024-03-28 18:35:35.491089315 +0000 UTC m=+0.368718672 container start 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 ceph-mds[66879]: -- 172.21.15.192:0/1113317306 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639c4b9da40 con 0x5639c4b71000 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: starting mds.i at 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 bash[66860]: 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 2024-03-28T18:35:35.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:35 smithi192 systemd[1]: Started Ceph mds.i for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:35.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:35 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2984207271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:36.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:35 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2984207271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:36.569 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:boot 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: fsmap 9 up:standby 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:36.570 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:boot 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: fsmap 9 up:standby 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:36.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:boot 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: fsmap 9 up:standby 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:37.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:37.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 -- 172.21.15.145:0/3014186510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 msgr2=0x7fde84108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:37.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 --2- 172.21.15.145:0/3014186510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde84108a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009ca0 tx=0x7fde7c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 -- 172.21.15.145:0/3014186510 shutdown_connections 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 --2- 172.21.15.145:0/3014186510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde84108a00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 --2- 172.21.15.145:0/3014186510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 0x7fde84101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 --2- 172.21.15.145:0/3014186510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde841009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 -- 172.21.15.145:0/3014186510 >> 172.21.15.145:0/3014186510 conn(0x7fde840fbb30 msgr2=0x7fde840fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 -- 172.21.15.145:0/3014186510 shutdown_connections 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.125+0000 7fde89fd4640 1 -- 172.21.15.145:0/3014186510 wait complete. 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 Processor -- start 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 -- start start 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde84069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 0x7fde8419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c220 con 0x7fde84102170 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c0a0 con 0x7fde841017b0 2024-03-28T18:35:37.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde89fd4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410c3a0 con 0x7fde841005c0 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde83fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde83fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54420/0 (socket says 172.21.15.145:54420) 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.126+0000 7fde83fff640 1 -- 172.21.15.145:0/1573706516 learned_addr learned my addr 172.21.15.145:0/1573706516 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde82ffd640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 0x7fde8419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde837fe640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde84069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 -- 172.21.15.145:0/1573706516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 msgr2=0x7fde84069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde84069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 -- 172.21.15.145:0/1573706516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 msgr2=0x7fde8419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 0x7fde8419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 -- 172.21.15.145:0/1573706516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8419f590 con 0x7fde84102170 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde837fe640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde84069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde83fff640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009bf0 tx=0x7fde7c035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c039040 con 0x7fde84102170 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde7c044070 con 0x7fde84102170 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8419f820 con 0x7fde84102170 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0043a0 con 0x7fde84102170 2024-03-28T18:35:37.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.127+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8419fd10 con 0x7fde84102170 2024-03-28T18:35:37.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.128+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde841078e0 con 0x7fde84102170 2024-03-28T18:35:37.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.129+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fde7c003ae0 con 0x7fde84102170 2024-03-28T18:35:37.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.130+0000 7fde80ff9640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:37.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.130+0000 7fde837fe640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:37.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.130+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fde7c0f0cc0 con 0x7fde84102170 2024-03-28T18:35:37.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.130+0000 7fde837fe640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde84101610 tx=0x7fde74005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:37.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.134+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fde7c0be200 con 0x7fde84102170 2024-03-28T18:35:37.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.352+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8402d250 con 0x7fde84102170 2024-03-28T18:35:37.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.353+0000 7fde80ff9640 1 -- 172.21.15.145:0/1573706516 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7682 (secure 0 0 0) 0x7fde7c0bdb00 con 0x7fde84102170 2024-03-28T18:35:37.353 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:37.353 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:35:37.353 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 10 2024-03-28T18:35:37.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 msgr2=0x7fde6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:37.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde84101610 tx=0x7fde74005d30 comp rx=0 tx=0).stop 2024-03-28T18:35:37.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 msgr2=0x7fde8419ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:37.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009bf0 tx=0x7fde7c035ae0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 shutdown_connections 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde84102170 0x7fde8419ecd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde841017b0 0x7fde8419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fde6c0a2800 0x7fde6c0a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 --2- 172.21.15.145:0/1573706516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde841005c0 0x7fde84069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 >> 172.21.15.145:0/1573706516 conn(0x7fde840fbb30 msgr2=0x7fde84104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 shutdown_connections 2024-03-28T18:35:37.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:37.355+0000 7fde89fd4640 1 -- 172.21.15.145:0/1573706516 wait complete. 2024-03-28T18:35:37.761 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:37 smithi145 ceph-mon[32151]: Deploying daemon mds.g on smithi145 2024-03-28T18:35:37.762 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1573706516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:37 smithi192 ceph-mon[36589]: Deploying daemon mds.g on smithi145 2024-03-28T18:35:37.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1573706516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:37.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:37 smithi164 ceph-mon[36248]: Deploying daemon mds.g on smithi145 2024-03-28T18:35:37.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1573706516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:38.026 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:37 smithi145 systemd[1]: Starting Ceph mds.g for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:38.464 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 podman[70658]: 2024-03-28 18:35:38.062215942 +0000 UTC m=+0.017588509 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:38.464 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 podman[70658]: 2024-03-28 18:35:38.17881686 +0000 UTC m=+0.134189416 container create bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T18:35:38.464 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 podman[70658]: 2024-03-28 18:35:38.464964657 +0000 UTC m=+0.420337209 container init bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:38.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:38 smithi145 ceph-mon[32151]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:38.776 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 podman[70658]: 2024-03-28 18:35:38.4677819 +0000 UTC m=+0.423154461 container start bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T18:35:38.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 ceph-mds[70677]: -- 172.21.15.145:0/4270870260 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56176ef33a40 con 0x56176ef06800 2024-03-28T18:35:38.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: starting mds.g at 2024-03-28T18:35:38.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 bash[70658]: bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 2024-03-28T18:35:38.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:35:38 smithi145 systemd[1]: Started Ceph mds.g for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:38 smithi192 ceph-mon[36589]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:38 smithi164 ceph-mon[36248]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:39.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 -- 172.21.15.145:0/2780028268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01032a0 msgr2=0x7fbcb0109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/2780028268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01032a0 0x7fbcb0109b30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009ca0 tx=0x7fbca802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 -- 172.21.15.145:0/2780028268 shutdown_connections 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/2780028268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01032a0 0x7fbcb0109b30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/2780028268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01028e0 0x7fbcb0102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/2780028268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 0x7fbcb0101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 -- 172.21.15.145:0/2780028268 >> 172.21.15.145:0/2780028268 conn(0x7fbcb00fcc40 msgr2=0x7fbcb00ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 -- 172.21.15.145:0/2780028268 shutdown_connections 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.475+0000 7fbcb73d4640 1 -- 172.21.15.145:0/2780028268 wait complete. 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 Processor -- start 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 -- start start 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 0x7fbcb0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 0x7fbcb019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb010d460 con 0x7fbcb01028e0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb010d2e0 con 0x7fbcb01016f0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb73d4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb010d5e0 con 0x7fbcb01032a0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54446/0 (socket says 172.21.15.145:54446) 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 -- 172.21.15.145:0/1807712310 learned_addr learned my addr 172.21.15.145:0/1807712310 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb5170640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 0x7fbcb0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb5971640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 0x7fbcb019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 -- 172.21.15.145:0/1807712310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 msgr2=0x7fbcb019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 0x7fbcb019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 -- 172.21.15.145:0/1807712310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 msgr2=0x7fbcb0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 0x7fbcb0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb496f640 1 -- 172.21.15.145:0/1807712310 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb019f6a0 con 0x7fbcb01028e0 2024-03-28T18:35:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.476+0000 7fbcb5971640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 0x7fbcb019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbcb496f640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b4a0 tx=0x7fbca000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0014070 con 0x7fbcb01028e0 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbca0010af0 con 0x7fbcb01028e0 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb019f990 con 0x7fbcb01028e0 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0017d10 con 0x7fbcb01028e0 2024-03-28T18:35:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.477+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb019ff10 con 0x7fbcb01028e0 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.478+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb0108a10 con 0x7fbcb01028e0 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.479+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbca0017070 con 0x7fbcb01028e0 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.479+0000 7fbca67fc640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 0x7fbc9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.480+0000 7fbcb5170640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 0x7fbc9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.480+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbca00cda00 con 0x7fbcb01028e0 2024-03-28T18:35:39.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.480+0000 7fbcb5170640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 0x7fbc9c0a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0102740 tx=0x7fbcac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:39.484 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.484+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbca009af40 con 0x7fbcb01028e0 2024-03-28T18:35:39.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.705+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcb002d070 con 0x7fbcb01028e0 2024-03-28T18:35:39.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.706+0000 7fbca67fc640 1 -- 172.21.15.145:0/1807712310 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8477 (secure 0 0 0) 0x7fbca009a840 con 0x7fbcb01028e0 2024-03-28T18:35:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:39.706 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:35:39.706 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 11 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 msgr2=0x7fbc9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 0x7fbc9c0a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0102740 tx=0x7fbcac008040 comp rx=0 tx=0).stop 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 msgr2=0x7fbcb0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b4a0 tx=0x7fbca000b960 comp rx=0 tx=0).stop 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 shutdown_connections 2024-03-28T18:35:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbc9c0a2800 0x7fbc9c0a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb01032a0 0x7fbcb019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb01028e0 0x7fbcb0197fc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 --2- 172.21.15.145:0/1807712310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb01016f0 0x7fbcb0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 >> 172.21.15.145:0/1807712310 conn(0x7fbcb00fcc40 msgr2=0x7fbcb0105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 shutdown_connections 2024-03-28T18:35:39.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:39.708+0000 7fbcb73d4640 1 -- 172.21.15.145:0/1807712310 wait complete. 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6838/1703430127,v1:172.21.15.145:6839/1703430127] up:boot 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: fsmap 10 up:standby 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:39.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6838/1703430127,v1:172.21.15.145:6839/1703430127] up:boot 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: fsmap 10 up:standby 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:39.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:39.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6838/1703430127,v1:172.21.15.145:6839/1703430127] up:boot 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: fsmap 10 up:standby 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:39.998 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:40.602 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:40 smithi145 ceph-mon[32151]: Deploying daemon mds.a on smithi145 2024-03-28T18:35:40.602 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:40 smithi145 ceph-mon[32151]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:40.602 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:40 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1807712310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:40 smithi192 ceph-mon[36589]: Deploying daemon mds.a on smithi145 2024-03-28T18:35:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:40 smithi192 ceph-mon[36589]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:40 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1807712310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:40 smithi164 ceph-mon[36248]: Deploying daemon mds.a on smithi145 2024-03-28T18:35:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:40 smithi164 ceph-mon[36248]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:40 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1807712310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:41.011 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:40 smithi145 systemd[1]: Starting Ceph mds.a for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:41.276 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 podman[71009]: 2024-03-28 18:35:40.914759896 +0000 UTC m=+0.018187330 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:41.276 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 podman[71009]: 2024-03-28 18:35:41.035585305 +0000 UTC m=+0.139012780 container create f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T18:35:41.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:41.746 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 podman[71009]: 2024-03-28 18:35:41.31509621 +0000 UTC m=+0.418523649 container init f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:35:41.746 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 podman[71009]: 2024-03-28 18:35:41.31783556 +0000 UTC m=+0.421262993 container start f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-28T18:35:41.746 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 ceph-mds[71028]: -- 172.21.15.145:0/1076924219 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563180eb3a40 con 0x563180e86c00 2024-03-28T18:35:41.747 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: starting mds.a at 2024-03-28T18:35:41.747 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 bash[71009]: f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 2024-03-28T18:35:41.747 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:35:41 smithi145 systemd[1]: Started Ceph mds.a for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- 172.21.15.145:0/3255619515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 msgr2=0x7f969c109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 --2- 172.21.15.145:0/3255619515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c109b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9694009ca0 tx=0x7f969402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- 172.21.15.145:0/3255619515 shutdown_connections 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 --2- 172.21.15.145:0/3255619515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c109b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 --2- 172.21.15.145:0/3255619515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1028b0 0x7f969c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 --2- 172.21.15.145:0/3255619515 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c1016c0 0x7f969c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- 172.21.15.145:0/3255619515 >> 172.21.15.145:0/3255619515 conn(0x7f969c0fcc50 msgr2=0x7f969c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- 172.21.15.145:0/3255619515 shutdown_connections 2024-03-28T18:35:41.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- 172.21.15.145:0/3255619515 wait complete. 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 Processor -- start 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.826+0000 7f96a4614640 1 -- start start 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1016c0 0x7f969c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c1028b0 0x7f969c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c10d340 con 0x7f969c103270 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c10d1c0 con 0x7f969c1016c0 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c10d4c0 con 0x7f969c1028b0 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54478/0 (socket says 172.21.15.145:54478) 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 -- 172.21.15.145:0/2601153751 learned_addr learned my addr 172.21.15.145:0/2601153751 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a23b0640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1016c0 0x7f969c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 -- 172.21.15.145:0/2601153751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c1028b0 msgr2=0x7f969c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:41.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c1028b0 0x7f969c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 -- 172.21.15.145:0/2601153751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1016c0 msgr2=0x7f969c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1016c0 0x7f969c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 -- 172.21.15.145:0/2601153751 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c19f580 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a2bb1640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9694037040 tx=0x7f9694032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694041020 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9694034d90 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c19f810 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694039040 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.827+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c19fd00 con 0x7f969c103270 2024-03-28T18:35:41.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.828+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c1089e0 con 0x7f969c103270 2024-03-28T18:35:41.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.829+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9694044020 con 0x7f969c103270 2024-03-28T18:35:41.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.830+0000 7f96937fe640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:41.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.830+0000 7f96a23b0640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:41.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.830+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96940f0960 con 0x7f969c103270 2024-03-28T18:35:41.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.830+0000 7f96a23b0640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 0x7f96880a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9698005fd0 tx=0x7f969800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:41.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:41.833+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f969404b020 con 0x7f969c103270 2024-03-28T18:35:42.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.047+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f969c02d190 con 0x7f969c103270 2024-03-28T18:35:42.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.048+0000 7f96937fe640 1 -- 172.21.15.145:0/2601153751 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9272 (secure 0 0 0) 0x7f96940bdea0 con 0x7f969c103270 2024-03-28T18:35:42.048 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:42.048 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:35:42.048 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 12 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 msgr2=0x7f96880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 0x7f96880a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9698005fd0 tx=0x7f969800b390 comp rx=0 tx=0).stop 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 msgr2=0x7f969c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9694037040 tx=0x7f9694032f20 comp rx=0 tx=0).stop 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 shutdown_connections 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f969c103270 0x7f969c19ef10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96880a2800 0x7f96880a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:42.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f969c1028b0 0x7f969c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:42.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 --2- 172.21.15.145:0/2601153751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f969c1016c0 0x7f969c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:42.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 >> 172.21.15.145:0/2601153751 conn(0x7f969c0fcc50 msgr2=0x7f969c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:42.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 shutdown_connections 2024-03-28T18:35:42.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:42.050+0000 7f96a4614640 1 -- 172.21.15.145:0/2601153751 wait complete. 2024-03-28T18:35:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: Deploying daemon mds.b on smithi164 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:boot 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: fsmap 11 up:standby 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:35:42.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:42 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2601153751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: Deploying daemon mds.b on smithi164 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:boot 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: fsmap 11 up:standby 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:35:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:42 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2601153751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: Deploying daemon mds.b on smithi164 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:boot 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: fsmap 11 up:standby 2024-03-28T18:35:42.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:35:42.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:42 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2601153751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:43.959 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:43 smithi164 systemd[1]: Starting Ceph mds.b for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:35:44.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.176+0000 7f2677ee8640 1 -- 172.21.15.145:0/262862081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 msgr2=0x7f26701009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.176+0000 7f2677ee8640 1 --2- 172.21.15.145:0/262862081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701009e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f266c009900 tx=0x7f266c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- 172.21.15.145:0/262862081 shutdown_connections 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- 172.21.15.145:0/262862081 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 0x7f2670108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- 172.21.15.145:0/262862081 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 0x7f2670101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- 172.21.15.145:0/262862081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701009e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- 172.21.15.145:0/262862081 >> 172.21.15.145:0/262862081 conn(0x7f26700fbb30 msgr2=0x7f26700fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- 172.21.15.145:0/262862081 shutdown_connections 2024-03-28T18:35:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- 172.21.15.145:0/262862081 wait complete. 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 Processor -- start 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- start start 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 0x7f2670196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 0x7f267019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267010c290 con 0x7f26701005c0 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267010c110 con 0x7f26701017b0 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2677ee8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267010c410 con 0x7f2670102170 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2675c84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2675c84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36806/0 (socket says 172.21.15.145:36806) 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.177+0000 7f2675c84640 1 -- 172.21.15.145:0/3449747859 learned_addr learned my addr 172.21.15.145:0/3449747859 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675483640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 0x7f2670196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2676485640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 0x7f267019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 -- 172.21.15.145:0/3449747859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 msgr2=0x7f267019de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 0x7f267019de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 -- 172.21.15.145:0/3449747859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 msgr2=0x7f2670196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 0x7f2670196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 -- 172.21.15.145:0/3449747859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267019e500 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2675c84640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f266c000c00 tx=0x7f266c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266c043020 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f266c0054e0 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267019e790 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266c003ce0 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.178+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267019ecb0 con 0x7f26701005c0 2024-03-28T18:35:44.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.179+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26701078e0 con 0x7f26701005c0 2024-03-28T18:35:44.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.180+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f266c046020 con 0x7f26701005c0 2024-03-28T18:35:44.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.181+0000 7f2666ffd640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 0x7f265c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:44.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.181+0000 7f2675483640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 0x7f265c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:44.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.181+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f266c0f0770 con 0x7f26701005c0 2024-03-28T18:35:44.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.181+0000 7f2675483640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 0x7f265c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2670197df0 tx=0x7f2660005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:44.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.185+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f266c04d020 con 0x7f26701005c0 2024-03-28T18:35:44.212 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:43 smithi164 podman[66793]: 2024-03-28 18:35:43.95915319 +0000 UTC m=+0.103400969 container create d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:35:44.212 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:43 smithi164 podman[66793]: 2024-03-28 18:35:43.873442028 +0000 UTC m=+0.017689774 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:35:44.405 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.405+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f267002d290 con 0x7f26701005c0 2024-03-28T18:35:44.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.406+0000 7f2666ffd640 1 -- 172.21.15.145:0/3449747859 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9272 (secure 0 0 0) 0x7f266c0bdcb0 con 0x7f26701005c0 2024-03-28T18:35:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:44.406 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:35:44.406 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 12 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 msgr2=0x7f265c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 0x7f265c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2670197df0 tx=0x7f2660005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 msgr2=0x7f26701968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f266c000c00 tx=0x7f266c02c6c0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 shutdown_connections 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f265c0a2800 0x7f265c0a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2670102170 0x7f267019de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26701017b0 0x7f2670196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 --2- 172.21.15.145:0/3449747859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26701005c0 0x7f26701968e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.408 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 >> 172.21.15.145:0/3449747859 conn(0x7f26700fbb30 msgr2=0x7f2670104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:44.409 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 shutdown_connections 2024-03-28T18:35:44.409 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:44.408+0000 7f2677ee8640 1 -- 172.21.15.145:0/3449747859 wait complete. 2024-03-28T18:35:44.480 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 podman[66793]: 2024-03-28 18:35:44.220398731 +0000 UTC m=+0.364646476 container init d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T18:35:44.481 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 podman[66793]: 2024-03-28 18:35:44.223794798 +0000 UTC m=+0.368042536 container start d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:35:44.481 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mds[66813]: -- 172.21.15.164:0/4171593943 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb5d51da40 con 0x55fb5d4f1000 2024-03-28T18:35:44.481 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: starting mds.b at 2024-03-28T18:35:44.481 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 bash[66793]: d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec 2024-03-28T18:35:44.481 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:35:44 smithi164 systemd[1]: Started Ceph mds.b for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:35:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3449747859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:44 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3449747859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:44 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3449747859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:44 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:45.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:45 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:boot 2024-03-28T18:35:45.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:45 smithi164 ceph-mon[36248]: fsmap 12 up:standby 2024-03-28T18:35:45.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:35:45.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:45 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:45.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:45 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:boot 2024-03-28T18:35:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:45 smithi192 ceph-mon[36589]: fsmap 12 up:standby 2024-03-28T18:35:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:35:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:45 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:45.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:45 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:boot 2024-03-28T18:35:45.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:45 smithi145 ceph-mon[32151]: fsmap 12 up:standby 2024-03-28T18:35:45.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:35:45.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:45 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:46.420 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 -- 172.21.15.145:0/4046907613 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 msgr2=0x7f8b241097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 --2- 172.21.15.145:0/4046907613 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b241097f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c009ca0 tx=0x7f8b1c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 -- 172.21.15.145:0/4046907613 shutdown_connections 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 --2- 172.21.15.145:0/4046907613 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b241097f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 --2- 172.21.15.145:0/4046907613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 --2- 172.21.15.145:0/4046907613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 0x7f8b24104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 -- 172.21.15.145:0/4046907613 >> 172.21.15.145:0/4046907613 conn(0x7f8b240fbb30 msgr2=0x7f8b240fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:46.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 -- 172.21.15.145:0/4046907613 shutdown_connections 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 -- 172.21.15.145:0/4046907613 wait complete. 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.527+0000 7f8b29654640 1 Processor -- start 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 -- start start 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 0x7f8b241979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2410d360 con 0x7f8b24107400 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2410d1e0 con 0x7f8b241021a0 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b29654640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2410d4e0 con 0x7f8b24104ad0 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b237fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b237fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36812/0 (socket says 172.21.15.145:36812) 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b237fe640 1 -- 172.21.15.145:0/1604457989 learned_addr learned my addr 172.21.15.145:0/1604457989 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:46.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b22ffd640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 0x7f8b241979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 -- 172.21.15.145:0/1604457989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 msgr2=0x7f8b241979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 0x7f8b241979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 -- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 msgr2=0x7f8b2419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 -- 172.21.15.145:0/1604457989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2419f5f0 con 0x7f8b24104ad0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b237fe640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b227fc640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24197f10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400e9c0 tx=0x7f8b1400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400ccc0 con 0x7f8b24104ad0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b14005020 con 0x7f8b24104ad0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.528+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14004a00 con 0x7f8b24104ad0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.529+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2419f8e0 con 0x7f8b24104ad0 2024-03-28T18:35:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.529+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2419fde0 con 0x7f8b24104ad0 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.529+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b240fd620 con 0x7f8b24104ad0 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.531+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8b140051d0 con 0x7f8b24104ad0 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.532+0000 7f8b0bfff640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 0x7f8b0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.532+0000 7f8b22ffd640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 0x7f8b0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.532+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b140ce500 con 0x7f8b24104ad0 2024-03-28T18:35:46.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.532+0000 7f8b22ffd640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 0x7f8b0c0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800bbe0 tx=0x7f8b1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:46.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.535+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b1409ba40 con 0x7f8b24104ad0 2024-03-28T18:35:46.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.757+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b2402d070 con 0x7f8b24104ad0 2024-03-28T18:35:46.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.757+0000 7f8b0bfff640 1 -- 172.21.15.145:0/1604457989 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10064 (secure 0 0 0) 0x7f8b1409b340 con 0x7f8b24104ad0 2024-03-28T18:35:46.757 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:46.758 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:35:46.758 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 13 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 msgr2=0x7f8b0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 0x7f8b0c0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800bbe0 tx=0x7f8b1800b390 comp rx=0 tx=0).stop 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 msgr2=0x7f8b24197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24197f10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400e9c0 tx=0x7f8b1400ee80 comp rx=0 tx=0).stop 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 shutdown_connections 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b24107400 0x7f8b2419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b0c0a2800 0x7f8b0c0a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b24104ad0 0x7f8b24197f10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.760 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 --2- 172.21.15.145:0/1604457989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b241021a0 0x7f8b241979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.761 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 >> 172.21.15.145:0/1604457989 conn(0x7f8b240fbb30 msgr2=0x7f8b24102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:46.761 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 shutdown_connections 2024-03-28T18:35:46.761 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.760+0000 7f8b29654640 1 -- 172.21.15.145:0/1604457989 wait complete. 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.770 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:46 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:35:46.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.878+0000 7fabdb461640 1 -- 172.21.15.145:0/2331451427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 msgr2=0x7fabd41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.878+0000 7fabdb461640 1 --2- 172.21.15.145:0/2331451427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd41097f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009ca0 tx=0x7fabcc02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 -- 172.21.15.145:0/2331451427 shutdown_connections 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 --2- 172.21.15.145:0/2331451427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd41097f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 --2- 172.21.15.145:0/2331451427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4104a70 0x7fabd4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 --2- 172.21.15.145:0/2331451427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4102140 0x7fabd4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 -- 172.21.15.145:0/2331451427 >> 172.21.15.145:0/2331451427 conn(0x7fabd40fbb30 msgr2=0x7fabd40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 -- 172.21.15.145:0/2331451427 shutdown_connections 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 -- 172.21.15.145:0/2331451427 wait complete. 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 Processor -- start 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.879+0000 7fabdb461640 1 -- start start 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4102140 0x7fabd4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4104a70 0x7fabd4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd410d3b0 con 0x7fabd41073a0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd410d230 con 0x7fabd4102140 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd410d530 con 0x7fabd4104a70 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36830/0 (socket says 172.21.15.145:36830) 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 -- 172.21.15.145:0/315192748 learned_addr learned my addr 172.21.15.145:0/315192748 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd91fd640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4102140 0x7fabd4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd89fc640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4104a70 0x7fabd4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 -- 172.21.15.145:0/315192748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4104a70 msgr2=0x7fabd4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4104a70 0x7fabd4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 -- 172.21.15.145:0/315192748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4102140 msgr2=0x7fabd4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4102140 0x7fabd4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 -- 172.21.15.145:0/315192748 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd419f5f0 con 0x7fabd41073a0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabd99fe640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabcc037040 tx=0x7fabcc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc041020 con 0x7fabd41073a0 2024-03-28T18:35:46.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabcc034cd0 con 0x7fabd41073a0 2024-03-28T18:35:46.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd419f880 con 0x7fabd41073a0 2024-03-28T18:35:46.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc039040 con 0x7fabd41073a0 2024-03-28T18:35:46.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.880+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd419fdc0 con 0x7fabd41073a0 2024-03-28T18:35:46.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:46.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:46 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.881+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd40fd620 con 0x7fabd41073a0 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.883+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fabcc044020 con 0x7fabd41073a0 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.884+0000 7fabca7fc640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 0x7fabc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.884+0000 7fabd91fd640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 0x7fabc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.884+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fabcc0f0890 con 0x7fabd41073a0 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.885+0000 7fabd91fd640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 0x7fabc00a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009990 tx=0x7fabd0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:46.887+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fabcc04b020 con 0x7fabd41073a0 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:46 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:47.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.079+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabd402d270 con 0x7fabd41073a0 2024-03-28T18:35:47.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.080+0000 7fabca7fc640 1 -- 172.21.15.145:0/315192748 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fabcc0bddd0 con 0x7fabd41073a0 2024-03-28T18:35:47.080 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:47.080 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":83,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:00.757071+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:47.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 msgr2=0x7fabc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 0x7fabc00a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009990 tx=0x7fabd0009110 comp rx=0 tx=0).stop 2024-03-28T18:35:47.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 msgr2=0x7fabd419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fabcc037040 tx=0x7fabcc032f20 comp rx=0 tx=0).stop 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 shutdown_connections 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabd41073a0 0x7fabd419ef80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fabc00a2880 0x7fabc00a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fabd4104a70 0x7fabd4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 --2- 172.21.15.145:0/315192748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabd4102140 0x7fabd4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 >> 172.21.15.145:0/315192748 conn(0x7fabd40fbb30 msgr2=0x7fabd4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 shutdown_connections 2024-03-28T18:35:47.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.082+0000 7fabdb461640 1 -- 172.21.15.145:0/315192748 wait complete. 2024-03-28T18:35:47.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:35:47.097 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:35:47.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 -- 172.21.15.145:0/1737697710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 msgr2=0x7f56301097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 --2- 172.21.15.145:0/1737697710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f56301097d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5628009ca0 tx=0x7f562802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 -- 172.21.15.145:0/1737697710 shutdown_connections 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 --2- 172.21.15.145:0/1737697710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f56301097d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 --2- 172.21.15.145:0/1737697710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 0x7f5630106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 --2- 172.21.15.145:0/1737697710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 0x7f5630104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 -- 172.21.15.145:0/1737697710 >> 172.21.15.145:0/1737697710 conn(0x7f56300fbb30 msgr2=0x7f56300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 -- 172.21.15.145:0/1737697710 shutdown_connections 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.205+0000 7f5637636640 1 -- 172.21.15.145:0/1737697710 wait complete. 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 Processor -- start 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 -- start start 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 0x7f56301979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 0x7f5630197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563010d320 con 0x7f56301073e0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563010d1a0 con 0x7f5630104ab0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5637636640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563010d4a0 con 0x7f5630102180 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5635bd3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5635bd3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36848/0 (socket says 172.21.15.145:36848) 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5635bd3640 1 -- 172.21.15.145:0/995674149 learned_addr learned my addr 172.21.15.145:0/995674149 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f5634bd1640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 0x7f5630197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.206+0000 7f56353d2640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 0x7f56301979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 -- 172.21.15.145:0/995674149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 msgr2=0x7f56301979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 0x7f56301979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 -- 172.21.15.145:0/995674149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 msgr2=0x7f5630197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 0x7f5630197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 -- 172.21.15.145:0/995674149 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563019f5c0 con 0x7f56301073e0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5635bd3640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5628037040 tx=0x7f5628032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628041020 con 0x7f56301073e0 2024-03-28T18:35:47.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5628034cd0 con 0x7f56301073e0 2024-03-28T18:35:47.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563019f850 con 0x7f56301073e0 2024-03-28T18:35:47.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628039040 con 0x7f56301073e0 2024-03-28T18:35:47.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.207+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563019fd40 con 0x7f56301073e0 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.208+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56300fd620 con 0x7f56301073e0 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.209+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5628044020 con 0x7f56301073e0 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.209+0000 7f56267fc640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 0x7f561c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.210+0000 7f56353d2640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 0x7f561c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.210+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56280f0460 con 0x7f56301073e0 2024-03-28T18:35:47.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.210+0000 7f56353d2640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 0x7f561c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f562c009bd0 tx=0x7f562c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.214 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.214+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f562804b020 con 0x7f56301073e0 2024-03-28T18:35:47.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.396+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56300622a0 con 0x7f56301073e0 2024-03-28T18:35:47.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.397+0000 7f56267fc640 1 -- 172.21.15.145:0/995674149 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56280bda30 con 0x7f56301073e0 2024-03-28T18:35:47.397 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 msgr2=0x7f561c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 0x7f561c0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f562c009bd0 tx=0x7f562c005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 msgr2=0x7f563019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5628037040 tx=0x7f5628032f40 comp rx=0 tx=0).stop 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 shutdown_connections 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56301073e0 0x7f563019ef50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5630104ab0 0x7f5630197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f561c0a2800 0x7f561c0a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 --2- 172.21.15.145:0/995674149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5630102180 0x7f56301979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 >> 172.21.15.145:0/995674149 conn(0x7f56300fbb30 msgr2=0x7f5630102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 shutdown_connections 2024-03-28T18:35:47.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.399+0000 7f5637636640 1 -- 172.21.15.145:0/995674149 wait complete. 2024-03-28T18:35:47.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 -- 172.21.15.145:0/1742579245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 msgr2=0x7ff888101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 --2- 172.21.15.145:0/1742579245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff888101af0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff884009900 tx=0x7ff88402bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 -- 172.21.15.145:0/1742579245 shutdown_connections 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 --2- 172.21.15.145:0/1742579245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 0x7ff888109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 --2- 172.21.15.145:0/1742579245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 0x7ff888102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 --2- 172.21.15.145:0/1742579245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff888101af0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 -- 172.21.15.145:0/1742579245 >> 172.21.15.145:0/1742579245 conn(0x7ff8880fcc40 msgr2=0x7ff8880ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 -- 172.21.15.145:0/1742579245 shutdown_connections 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.522+0000 7ff8902dc640 1 -- 172.21.15.145:0/1742579245 wait complete. 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 Processor -- start 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 -- start start 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 0x7ff888197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 0x7ff88819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88810d340 con 0x7ff8881016d0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88810d1c0 con 0x7ff888103280 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88810d4c0 con 0x7ff8881028c0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36874/0 (socket says 172.21.15.145:36874) 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 -- 172.21.15.145:0/3439810913 learned_addr learned my addr 172.21.15.145:0/3439810913 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88d877640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 0x7ff888197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e879640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 0x7ff88819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 -- 172.21.15.145:0/3439810913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 msgr2=0x7ff888197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 0x7ff888197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 -- 172.21.15.145:0/3439810913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 msgr2=0x7ff88819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 0x7ff88819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 -- 172.21.15.145:0/3439810913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88819f610 con 0x7ff8881016d0 2024-03-28T18:35:47.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff88e078640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff884000c00 tx=0x7ff88402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884041070 con 0x7ff8881016d0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88819f8a0 con 0x7ff8881016d0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.523+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff884004450 con 0x7ff8881016d0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.524+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884034440 con 0x7ff8881016d0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.524+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88819fd90 con 0x7ff8881016d0 2024-03-28T18:35:47.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.524+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff888108a10 con 0x7ff8881016d0 2024-03-28T18:35:47.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.526+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff884044070 con 0x7ff8881016d0 2024-03-28T18:35:47.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.527+0000 7ff87f7fe640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 0x7ff8740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.527+0000 7ff88d877640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 0x7ff8740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.527+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8840f02b0 con 0x7ff8881016d0 2024-03-28T18:35:47.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.527+0000 7ff88d877640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 0x7ff8740a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff888198f00 tx=0x7ff878005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.530 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.530+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff8840bd210 con 0x7ff8881016d0 2024-03-28T18:35:47.717 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.715+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff88802d240 con 0x7ff8881016d0 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.716+0000 7ff87f7fe640 1 -- 172.21.15.145:0/3439810913 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff8840c12b0 con 0x7ff8881016d0 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout:[ 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout: "poolnum": 1, 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout: "poolname": ".mgr" 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:35:47.718 INFO:teuthology.orchestra.run.smithi145.stdout:] 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 msgr2=0x7ff8740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 0x7ff8740a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff888198f00 tx=0x7ff878005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 msgr2=0x7ff8881979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff884000c00 tx=0x7ff88402c640 comp rx=0 tx=0).stop 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 shutdown_connections 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff888103280 0x7ff88819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff8740a2800 0x7ff8740a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8881028c0 0x7ff888197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 --2- 172.21.15.145:0/3439810913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8881016d0 0x7ff8881979f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 >> 172.21.15.145:0/3439810913 conn(0x7ff8880fcc40 msgr2=0x7ff888105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 shutdown_connections 2024-03-28T18:35:47.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.718+0000 7ff8902dc640 1 -- 172.21.15.145:0/3439810913 wait complete. 2024-03-28T18:35:47.727 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1604457989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/315192748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.728 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:47 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/995674149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:47.729 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T18:35:47.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1604457989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/315192748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:47 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/995674149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1604457989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/315192748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:47.829 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:47 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/995674149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:35:47.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- 172.21.15.145:0/2614681304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff970101790 msgr2=0x7ff970101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- 172.21.15.145:0/2614681304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff970101790 0x7ff970101c10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff960009900 tx=0x7ff96002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:47.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- 172.21.15.145:0/2614681304 shutdown_connections 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- 172.21.15.145:0/2614681304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970102150 0x7ff9701089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- 172.21.15.145:0/2614681304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff970101790 0x7ff970101c10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- 172.21.15.145:0/2614681304 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9701005a0 0x7ff9701009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- 172.21.15.145:0/2614681304 >> 172.21.15.145:0/2614681304 conn(0x7ff9700fbb30 msgr2=0x7ff9700fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- 172.21.15.145:0/2614681304 shutdown_connections 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- 172.21.15.145:0/2614681304 wait complete. 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 Processor -- start 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- start start 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970101790 0x7ff97019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 0x7ff97019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010c270 con 0x7ff9701005a0 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010c0f0 con 0x7ff970102150 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.835+0000 7ff977a75640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010c3f0 con 0x7ff970101790 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36898/0 (socket says 172.21.15.145:36898) 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 -- 172.21.15.145:0/2544528876 learned_addr learned my addr 172.21.15.145:0/2544528876 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff976012640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 0x7ff97019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975010640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970101790 0x7ff97019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 -- 172.21.15.145:0/2544528876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970101790 msgr2=0x7ff97019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970101790 0x7ff97019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 -- 172.21.15.145:0/2544528876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 msgr2=0x7ff97019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 0x7ff97019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 -- 172.21.15.145:0/2544528876 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97019f660 con 0x7ff9701005a0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff976012640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 0x7ff97019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff975811640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00e9c0 tx=0x7ff96c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c00cde0 con 0x7ff9701005a0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97019f8f0 con 0x7ff9701005a0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96c010910 con 0x7ff9701005a0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c018a00 con 0x7ff9701005a0 2024-03-28T18:35:47.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.836+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97019fe10 con 0x7ff9701005a0 2024-03-28T18:35:47.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.837+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970107940 con 0x7ff9701005a0 2024-03-28T18:35:47.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.838+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff96c01f020 con 0x7ff9701005a0 2024-03-28T18:35:47.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.839+0000 7ff966ffd640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 0x7ff95c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:47.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.839+0000 7ff975010640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 0x7ff95c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:47.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.839+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff96c0ce7a0 con 0x7ff9701005a0 2024-03-28T18:35:47.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.840+0000 7ff975010640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 0x7ff95c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff96002c240 tx=0x7ff96002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:47.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:47.843+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff96c09bce0 con 0x7ff9701005a0 2024-03-28T18:35:48.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.036+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff97002d2e0 con 0x7ff9701005a0 2024-03-28T18:35:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:48 smithi145 ceph-mon[32151]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:48 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3439810913' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:35:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:48 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:35:48.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:48 smithi192 ceph-mon[36589]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:48 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3439810913' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:35:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:48 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:35:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:48.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.908+0000 7ff966ffd640 1 -- 172.21.15.145:0/2544528876 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff96c09b5e0 con 0x7ff9701005a0 2024-03-28T18:35:48.908 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_metadata' created 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 msgr2=0x7ff95c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 0x7ff95c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff96002c240 tx=0x7ff96002df90 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 msgr2=0x7ff970069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00e9c0 tx=0x7ff96c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 shutdown_connections 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff970102150 0x7ff97019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff95c0a2800 0x7ff95c0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970101790 0x7ff97019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 --2- 172.21.15.145:0/2544528876 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9701005a0 0x7ff970069370 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.911+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 >> 172.21.15.145:0/2544528876 conn(0x7ff9700fbb30 msgr2=0x7ff970104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.912+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 shutdown_connections 2024-03-28T18:35:48.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:48.912+0000 7ff977a75640 1 -- 172.21.15.145:0/2544528876 wait complete. 2024-03-28T18:35:48.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T18:35:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:48 smithi164 ceph-mon[36248]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:48 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3439810913' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:35:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:48 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:35:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 -- 172.21.15.145:0/2948035092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 msgr2=0x7f543c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 --2- 172.21.15.145:0/2948035092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c101c20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f542c009900 tx=0x7f542c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 -- 172.21.15.145:0/2948035092 shutdown_connections 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 --2- 172.21.15.145:0/2948035092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c102160 0x7f543c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 --2- 172.21.15.145:0/2948035092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c101c20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 --2- 172.21.15.145:0/2948035092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 0x7f543c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.034+0000 7f5444769640 1 -- 172.21.15.145:0/2948035092 >> 172.21.15.145:0/2948035092 conn(0x7f543c0fbb40 msgr2=0x7f543c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- 172.21.15.145:0/2948035092 shutdown_connections 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- 172.21.15.145:0/2948035092 wait complete. 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 Processor -- start 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- start start 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 0x7f543c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c102160 0x7f543c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c10c230 con 0x7f543c1017a0 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c10c0b0 con 0x7f543c1005b0 2024-03-28T18:35:49.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5444769640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c10c3b0 con 0x7f543c102160 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5441d04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5441d04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36920/0 (socket says 172.21.15.145:36920) 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5441d04640 1 -- 172.21.15.145:0/1648111180 learned_addr learned my addr 172.21.15.145:0/1648111180 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.035+0000 7f5442505640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 0x7f543c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 -- 172.21.15.145:0/1648111180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c102160 msgr2=0x7f543c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c102160 0x7f543c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 -- 172.21.15.145:0/1648111180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 msgr2=0x7f543c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 0x7f543c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 -- 172.21.15.145:0/1648111180 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c19f620 con 0x7f543c1017a0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5441d04640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f542c009230 tx=0x7f542c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c041070 con 0x7f543c1017a0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f542c0054e0 con 0x7f543c1017a0 2024-03-28T18:35:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c19f8b0 con 0x7f543c1017a0 2024-03-28T18:35:49.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c034440 con 0x7f543c1017a0 2024-03-28T18:35:49.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.036+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c19fda0 con 0x7f543c1017a0 2024-03-28T18:35:49.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.037+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c19ffa0 con 0x7f543c1017a0 2024-03-28T18:35:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.038+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f542c044070 con 0x7f543c1017a0 2024-03-28T18:35:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.039+0000 7f54337fe640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 0x7f54280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.039+0000 7f5442505640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 0x7f54280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.039+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f542c0f0440 con 0x7f543c1017a0 2024-03-28T18:35:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.039+0000 7f5442505640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 0x7f54280a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f543c101600 tx=0x7f543800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:49.043 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.043+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f542c0bd810 con 0x7f543c1017a0 2024-03-28T18:35:49.218 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.218+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f543c1a0170 con 0x7f543c1017a0 2024-03-28T18:35:49.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.915+0000 7f54337fe640 1 -- 172.21.15.145:0/1648111180 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) v1 ==== 171+0+0 (secure 0 0 0) 0x7f542c0bd110 con 0x7f543c1017a0 2024-03-28T18:35:49.915 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_data' created 2024-03-28T18:35:49.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.917+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 msgr2=0x7f54280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.917+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 0x7f54280a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f543c101600 tx=0x7f543800b390 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.917+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 msgr2=0x7f543c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.917+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f542c009230 tx=0x7f542c005dd0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 shutdown_connections 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f54280a2800 0x7f54280a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c102160 0x7f543c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f543c1017a0 0x7f543c197f40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 --2- 172.21.15.145:0/1648111180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f543c1005b0 0x7f543c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 >> 172.21.15.145:0/1648111180 conn(0x7f543c0fbb40 msgr2=0x7f543c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 shutdown_connections 2024-03-28T18:35:49.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:49.918+0000 7f5444769640 1 -- 172.21.15.145:0/1648111180 wait complete. 2024-03-28T18:35:49.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.937 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T18:35:49.938 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:49 smithi145 ceph-mon[32151]: pgmap v282: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T18:35:49.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:49 smithi164 ceph-mon[36248]: pgmap v282: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:50.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 -- 172.21.15.145:0/3556819688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc104ab0 msgr2=0x7faebc106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:50.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 --2- 172.21.15.145:0/3556819688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc104ab0 0x7faebc106ea0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009900 tx=0x7faeb002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:50.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 -- 172.21.15.145:0/3556819688 shutdown_connections 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 --2- 172.21.15.145:0/3556819688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc1073e0 0x7faebc1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 --2- 172.21.15.145:0/3556819688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc104ab0 0x7faebc106ea0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 --2- 172.21.15.145:0/3556819688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 0x7faebc104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 -- 172.21.15.145:0/3556819688 >> 172.21.15.145:0/3556819688 conn(0x7faebc0fbb30 msgr2=0x7faebc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 -- 172.21.15.145:0/3556819688 shutdown_connections 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.043+0000 7faec29c6640 1 -- 172.21.15.145:0/3556819688 wait complete. 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 Processor -- start 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 -- start start 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 0x7faebc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc104ab0 0x7faebc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d300 con 0x7faebc1073e0 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d180 con 0x7faebc102180 2024-03-28T18:35:50.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec29c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc10d480 con 0x7faebc104ab0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec0f63640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec0f63640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36940/0 (socket says 172.21.15.145:36940) 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faec0f63640 1 -- 172.21.15.145:0/2306529714 learned_addr learned my addr 172.21.15.145:0/2306529714 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faebbfff640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 0x7faebc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.044+0000 7faebb7fe640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc104ab0 0x7faebc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 -- 172.21.15.145:0/2306529714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc104ab0 msgr2=0x7faebc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc104ab0 0x7faebc197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 -- 172.21.15.145:0/2306529714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 msgr2=0x7faebc197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 0x7faebc197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 -- 172.21.15.145:0/2306529714 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc19f660 con 0x7faebc1073e0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec0f63640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faeb400eaa0 tx=0x7faeb400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb400cdb0 con 0x7faebc1073e0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc19f950 con 0x7faebc1073e0 2024-03-28T18:35:50.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faeb4005020 con 0x7faebc1073e0 2024-03-28T18:35:50.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40116b0 con 0x7faebc1073e0 2024-03-28T18:35:50.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.045+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc19fe20 con 0x7faebc1073e0 2024-03-28T18:35:50.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.046+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc0fd620 con 0x7faebc1073e0 2024-03-28T18:35:50.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.047+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faeb4005560 con 0x7faebc1073e0 2024-03-28T18:35:50.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.048+0000 7faeb97fa640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:50.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.048+0000 7faebbfff640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:50.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.048+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7879+0+0 (secure 0 0 0) 0x7faeb40cea60 con 0x7faebc1073e0 2024-03-28T18:35:50.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.048+0000 7faebbfff640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 0x7faea40a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faeac009bd0 tx=0x7faeac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:50.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.052+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faeb4016030 con 0x7faebc1073e0 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:35:50.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2544528876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:35:50.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:35:50.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T18:35:50.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:49 smithi192 ceph-mon[36589]: pgmap v282: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:50.264 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.264+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faebc02d290 con 0x7faebc1073e0 2024-03-28T18:35:50.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.916+0000 7faeb97fa640 1 -- 172.21.15.145:0/2306529714 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7faeb409b660 con 0x7faebc1073e0 2024-03-28T18:35:50.917 INFO:teuthology.orchestra.run.smithi145.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 msgr2=0x7faea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 0x7faea40a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faeac009bd0 tx=0x7faeac005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 msgr2=0x7faebc19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faeb400eaa0 tx=0x7faeb400ef60 comp rx=0 tx=0).stop 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 shutdown_connections 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faebc1073e0 0x7faebc19eff0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7faea40a2800 0x7faea40a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faebc104ab0 0x7faebc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 --2- 172.21.15.145:0/2306529714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faebc102180 0x7faebc197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:50.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 >> 172.21.15.145:0/2306529714 conn(0x7faebc0fbb30 msgr2=0x7faebc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:50.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 shutdown_connections 2024-03-28T18:35:50.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:50.919+0000 7faec29c6640 1 -- 172.21.15.145:0/2306529714 wait complete. 2024-03-28T18:35:50.933 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T18:35:50.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T18:35:50.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:50 smithi145 ceph-mon[32151]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:50.958 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:50 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T18:35:50.959 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:50 smithi145 ceph-mon[32151]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:35:50.959 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:50 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:35:50.959 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:50 smithi145 ceph-mon[32151]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T18:35:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:50 smithi164 ceph-mon[36248]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:50 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T18:35:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:50 smithi164 ceph-mon[36248]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:35:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:50 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:35:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:50 smithi164 ceph-mon[36248]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T18:35:51.044 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 -- 172.21.15.145:0/3669719986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca0104b10 msgr2=0x7f7ca0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.044 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/3669719986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca0104b10 0x7f7ca0106f00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c9002bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:51.044 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 -- 172.21.15.145:0/3669719986 shutdown_connections 2024-03-28T18:35:51.044 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/3669719986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0107440 0x7f7ca0109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.044 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/3669719986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca0104b10 0x7f7ca0106f00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/3669719986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca01021e0 0x7f7ca01045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 -- 172.21.15.145:0/3669719986 >> 172.21.15.145:0/3669719986 conn(0x7f7ca00fbb30 msgr2=0x7f7ca00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 -- 172.21.15.145:0/3669719986 shutdown_connections 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.042+0000 7f7ca6c85640 1 -- 172.21.15.145:0/3669719986 wait complete. 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 Processor -- start 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 -- start start 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0104b10 0x7f7ca0100190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 0x7f7ca01006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca010d320 con 0x7f7ca01021e0 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca010d1a0 con 0x7f7ca0104b10 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca6c85640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca010d4a0 con 0x7f7ca0107440 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36960/0 (socket says 172.21.15.145:36960) 2024-03-28T18:35:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 -- 172.21.15.145:0/1736473131 learned_addr learned my addr 172.21.15.145:0/1736473131 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7c97fff640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0104b10 0x7f7ca0100190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca5222640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 0x7f7ca01006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 -- 172.21.15.145:0/1736473131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 msgr2=0x7f7ca01006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 0x7f7ca01006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 -- 172.21.15.145:0/1736473131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0104b10 msgr2=0x7f7ca0100190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0104b10 0x7f7ca0100190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 -- 172.21.15.145:0/1736473131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca0100f60 con 0x7f7ca01021e0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca5222640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 0x7f7ca01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.043+0000 7f7ca4a21640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00d770 tx=0x7f7c9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.044+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c017070 con 0x7f7ca01021e0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.044+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c9c005500 con 0x7f7ca01021e0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.044+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca01a1b10 con 0x7f7ca01021e0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.044+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c004e00 con 0x7f7ca01021e0 2024-03-28T18:35:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.044+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca01a2030 con 0x7f7ca01021e0 2024-03-28T18:35:51.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.045+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca00fd620 con 0x7f7ca01021e0 2024-03-28T18:35:51.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.046+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7c9c004fa0 con 0x7f7ca01021e0 2024-03-28T18:35:51.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.047+0000 7f7c95ffb640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 0x7f7c880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.047+0000 7f7c97fff640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 0x7f7c880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.047+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7c9c0ce2b0 con 0x7f7ca01021e0 2024-03-28T18:35:51.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.047+0000 7f7c97fff640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 0x7f7c880a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c90000c00 tx=0x7f7c90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:51.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.051+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c9c09b4a0 con 0x7f7ca01021e0 2024-03-28T18:35:51.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:50 smithi192 ceph-mon[36589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:50 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1648111180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T18:35:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:50 smithi192 ceph-mon[36589]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:35:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:50 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:35:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:50 smithi192 ceph-mon[36589]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T18:35:51.236 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.236+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f7ca002d240 con 0x7f7ca01021e0 2024-03-28T18:35:51.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.688+0000 7f7c95ffb640 1 -- 172.21.15.145:0/1736473131 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) v1 ==== 157+0+0 (secure 0 0 0) 0x7f7c9c09ada0 con 0x7f7ca01021e0 2024-03-28T18:35:51.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 msgr2=0x7f7c880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 0x7f7c880a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c90000c00 tx=0x7f7c90005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 msgr2=0x7f7ca0101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00d770 tx=0x7f7c9c00dc30 comp rx=0 tx=0).stop 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 shutdown_connections 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c880a2880 0x7f7c880a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ca0107440 0x7f7ca01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ca0104b10 0x7f7ca0100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 --2- 172.21.15.145:0/1736473131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ca01021e0 0x7f7ca0101b50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 >> 172.21.15.145:0/1736473131 conn(0x7f7ca00fbb30 msgr2=0x7f7ca0102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 shutdown_connections 2024-03-28T18:35:51.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.691+0000 7f7ca6c85640 1 -- 172.21.15.145:0/1736473131 wait complete. 2024-03-28T18:35:51.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T18:35:51.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1653712730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec0ff9b0 msgr2=0x7f13ec0ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1653712730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec0ffdd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009930 tx=0x7f13dc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:51.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1653712730 shutdown_connections 2024-03-28T18:35:51.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1653712730 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec1033e0 0x7f13ec109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1653712730 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec102a20 0x7f13ec102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1653712730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec0ffdd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1653712730 >> 172.21.15.145:0/1653712730 conn(0x7f13ec0fb2c0 msgr2=0x7f13ec0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1653712730 shutdown_connections 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1653712730 wait complete. 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 Processor -- start 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.858+0000 7f13f1eb2640 1 -- start start 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec1033e0 0x7f13ec19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec10d390 con 0x7f13ec102a20 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec10d210 con 0x7f13ec0ff9b0 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec10d510 con 0x7f13ec1033e0 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36982/0 (socket says 172.21.15.145:36982) 2024-03-28T18:35:51.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 -- 172.21.15.145:0/1365898513 learned_addr learned my addr 172.21.15.145:0/1365898513 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13ebfff640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec1033e0 0x7f13ec19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eb7fe640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 -- 172.21.15.145:0/1365898513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec1033e0 msgr2=0x7f13ec19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec1033e0 0x7f13ec19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 -- 172.21.15.145:0/1365898513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 msgr2=0x7f13ec197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 -- 172.21.15.145:0/1365898513 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec19f620 con 0x7f13ec102a20 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eb7fe640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13eaffd640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13e000d950 tx=0x7f13e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e0017070 con 0x7f13ec102a20 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13e0009c30 con 0x7f13ec102a20 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec19f910 con 0x7f13ec102a20 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e000f6b0 con 0x7f13ec102a20 2024-03-28T18:35:51.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.859+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec19fe60 con 0x7f13ec102a20 2024-03-28T18:35:51.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.860+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec108b50 con 0x7f13ec102a20 2024-03-28T18:35:51.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.861+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f13e00109d0 con 0x7f13ec102a20 2024-03-28T18:35:51.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.862+0000 7f13e8ff9640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 0x7f13d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:51.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.862+0000 7f13eb7fe640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 0x7f13d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:51.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.862+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f13e00ce7f0 con 0x7f13ec102a20 2024-03-28T18:35:51.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.863+0000 7f13eb7fe640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 0x7f13d40a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0096c0 tx=0x7f13dc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:51.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:51.867+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13e0009de0 con 0x7f13ec102a20 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T18:35:51.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:51 smithi164 ceph-mon[36248]: pgmap v285: 97 pgs: 12 creating+peering, 68 unknown, 17 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T18:35:52.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:51 smithi145 ceph-mon[32151]: pgmap v285: 97 pgs: 12 creating+peering, 68 unknown, 17 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:52.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:52.051+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f13ec02d070 con 0x7f13ec102a20 2024-03-28T18:35:52.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:35:52.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2306529714' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:35:52.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T18:35:52.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T18:35:52.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T18:35:52.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:51 smithi192 ceph-mon[36589]: pgmap v285: 97 pgs: 12 creating+peering, 68 unknown, 17 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:active 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:52 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:active 2024-03-28T18:35:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:53.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:53.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:52 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:35:53.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1736473131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] up:active 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:52 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:35:54.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.003+0000 7f13e8ff9640 1 -- 172.21.15.145:0/1365898513 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) v1 ==== 228+0+0 (secure 0 0 0) 0x7f13e009b860 con 0x7f13ec102a20 2024-03-28T18:35:54.004 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_data_ec' created 2024-03-28T18:35:54.006 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 msgr2=0x7f13d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:54.006 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 0x7f13d40a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0096c0 tx=0x7f13dc02df90 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 msgr2=0x7f13ec197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13e000d950 tx=0x7f13e000de10 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 shutdown_connections 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13d40a2800 0x7f13d40a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13ec1033e0 0x7f13ec19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13ec102a20 0x7f13ec197f40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 --2- 172.21.15.145:0/1365898513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13ec0ff9b0 0x7f13ec197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.006+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 >> 172.21.15.145:0/1365898513 conn(0x7f13ec0fb2c0 msgr2=0x7f13ec105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.007+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 shutdown_connections 2024-03-28T18:35:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.007+0000 7f13f1eb2640 1 -- 172.21.15.145:0/1365898513 wait complete. 2024-03-28T18:35:54.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T18:35:54.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.128+0000 7fdb14b0c640 1 -- 172.21.15.145:0/2069979956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb101073a0 msgr2=0x7fdb101097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.128+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/2069979956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb101073a0 0x7fdb101097f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009ca0 tx=0x7fdb0802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- 172.21.15.145:0/2069979956 shutdown_connections 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/2069979956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb101073a0 0x7fdb101097f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/2069979956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb10104a70 0x7fdb10106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/2069979956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 0x7fdb10104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- 172.21.15.145:0/2069979956 >> 172.21.15.145:0/2069979956 conn(0x7fdb100fbb30 msgr2=0x7fdb100fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- 172.21.15.145:0/2069979956 shutdown_connections 2024-03-28T18:35:54.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- 172.21.15.145:0/2069979956 wait complete. 2024-03-28T18:35:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 Processor -- start 2024-03-28T18:35:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- start start 2024-03-28T18:35:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 0x7fdb10197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:54.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb101073a0 0x7fdb1019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1010d360 con 0x7fdb10104a70 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1010d1e0 con 0x7fdb101073a0 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.129+0000 7fdb14b0c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1010d4e0 con 0x7fdb10102140 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49904/0 (socket says 172.21.15.145:49904) 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 -- 172.21.15.145:0/3940708484 learned_addr learned my addr 172.21.15.145:0/3940708484 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0e59c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 0x7fdb10197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0ed9d640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb101073a0 0x7fdb1019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 -- 172.21.15.145:0/3940708484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 msgr2=0x7fdb10197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 0x7fdb10197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 -- 172.21.15.145:0/3940708484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb101073a0 msgr2=0x7fdb1019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb101073a0 0x7fdb1019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 -- 172.21.15.145:0/3940708484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1019f620 con 0x7fdb10104a70 2024-03-28T18:35:54.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb0dd9b640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400d770 tx=0x7fdb0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04017070 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb04009c30 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1019f910 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400f620 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.130+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1019fe60 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.131+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb100fd620 con 0x7fdb10104a70 2024-03-28T18:35:54.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.132+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdb0401a070 con 0x7fdb10104a70 2024-03-28T18:35:54.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.133+0000 7fdaff7fe640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 0x7fdaf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:54.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.133+0000 7fdb0e59c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 0x7fdaf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:54.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.133+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fdb040ce750 con 0x7fdb10104a70 2024-03-28T18:35:54.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.133+0000 7fdb0e59c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 0x7fdaf80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009bd0 tx=0x7fdb00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:54.136 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.136+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdb04009de0 con 0x7fdb10104a70 2024-03-28T18:35:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:54 smithi145 ceph-mon[32151]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:35:54.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:54 smithi145 ceph-mon[32151]: pgmap v288: 97 pgs: 12 creating+peering, 39 unknown, 46 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-28T18:35:54.310 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:54.310+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fdb10062310 con 0x7fdb10104a70 2024-03-28T18:35:54.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:54 smithi192 ceph-mon[36589]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:35:54.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:54 smithi192 ceph-mon[36589]: pgmap v288: 97 pgs: 12 creating+peering, 39 unknown, 46 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-28T18:35:54.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:54 smithi164 ceph-mon[36248]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:35:54.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:54 smithi164 ceph-mon[36248]: pgmap v288: 97 pgs: 12 creating+peering, 39 unknown, 46 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-28T18:35:55.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.005+0000 7fdaff7fe640 1 -- 172.21.15.145:0/3940708484 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdb0409b5d0 con 0x7fdb10104a70 2024-03-28T18:35:55.005 INFO:teuthology.orchestra.run.smithi145.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.009+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 msgr2=0x7fdaf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.009+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 0x7fdaf80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009bd0 tx=0x7fdb00005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.009+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 msgr2=0x7fdb10197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.009+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400d770 tx=0x7fdb0400dc30 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 shutdown_connections 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb101073a0 0x7fdb1019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb10104a70 0x7fdb10197f40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fdaf80a2800 0x7fdaf80a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 --2- 172.21.15.145:0/3940708484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb10102140 0x7fdb10197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 >> 172.21.15.145:0/3940708484 conn(0x7fdb100fbb30 msgr2=0x7fdb10102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 shutdown_connections 2024-03-28T18:35:55.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.010+0000 7fdb14b0c640 1 -- 172.21.15.145:0/3940708484 wait complete. 2024-03-28T18:35:55.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T18:35:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 -- 172.21.15.145:0/1052217723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848104b00 msgr2=0x7fa848106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 --2- 172.21.15.145:0/1052217723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848104b00 0x7fa848106ef0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009900 tx=0x7fa83c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 -- 172.21.15.145:0/1052217723 shutdown_connections 2024-03-28T18:35:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 --2- 172.21.15.145:0/1052217723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa848107430 0x7fa848109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 --2- 172.21.15.145:0/1052217723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848104b00 0x7fa848106ef0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 --2- 172.21.15.145:0/1052217723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8481021d0 0x7fa8481045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 -- 172.21.15.145:0/1052217723 >> 172.21.15.145:0/1052217723 conn(0x7fa8480fbb40 msgr2=0x7fa8480fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 -- 172.21.15.145:0/1052217723 shutdown_connections 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.141+0000 7fa84d019640 1 -- 172.21.15.145:0/1052217723 wait complete. 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 Processor -- start 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 -- start start 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 0x7fa848197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 0x7fa848197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84810d3d0 con 0x7fa848107430 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84810d250 con 0x7fa8481021d0 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84d019640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84810d550 con 0x7fa848104b00 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84759e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84759e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49928/0 (socket says 172.21.15.145:49928) 2024-03-28T18:35:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84759e640 1 -- 172.21.15.145:0/2352629721 learned_addr learned my addr 172.21.15.145:0/2352629721 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84659c640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 0x7fa848197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa846d9d640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 0x7fa848197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84759e640 1 -- 172.21.15.145:0/2352629721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 msgr2=0x7fa848197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.142+0000 7fa84759e640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 0x7fa848197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84759e640 1 -- 172.21.15.145:0/2352629721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 msgr2=0x7fa848197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84759e640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 0x7fa848197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84759e640 1 -- 172.21.15.145:0/2352629721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84819f660 con 0x7fa848107430 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84659c640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 0x7fa848197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa846d9d640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 0x7fa848197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84759e640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa84000b970 tx=0x7fa84000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840019070 con 0x7fa848107430 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84819f950 con 0x7fa848107430 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa84000cb40 con 0x7fa848107430 2024-03-28T18:35:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840007610 con 0x7fa848107430 2024-03-28T18:35:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.143+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84819fec0 con 0x7fa848107430 2024-03-28T18:35:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.144+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8480fd630 con 0x7fa848107430 2024-03-28T18:35:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.145+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa84001c070 con 0x7fa848107430 2024-03-28T18:35:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.145+0000 7fa82ffff640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 0x7fa8300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:55.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.145+0000 7fa846d9d640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 0x7fa8300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:55.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.145+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fa8400ce1a0 con 0x7fa848107430 2024-03-28T18:35:55.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.146+0000 7fa846d9d640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 0x7fa8300a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa838009c20 tx=0x7fa838005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:55.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.150+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa84009a9d0 con 0x7fa848107430 2024-03-28T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:55 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:55 smithi164 ceph-mon[36248]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:55 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T18:35:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:55 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:35:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:55 smithi145 ceph-mon[32151]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:35:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:55 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T18:35:55.369 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:55.369+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fa8480622a0 con 0x7fa848107430 2024-03-28T18:35:55.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:55 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1365898513' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:35:55.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:55 smithi192 ceph-mon[36589]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:35:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:55 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T18:35:55.906 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:35:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:35:55.553+0000 7f84ef9a3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:35:56.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.025+0000 7fa82ffff640 1 -- 172.21.15.145:0/2352629721 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa84009ea70 con 0x7fa848107430 2024-03-28T18:35:56.025 INFO:teuthology.orchestra.run.smithi145.stderr:added data pool 4 to fsmap 2024-03-28T18:35:56.027 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 msgr2=0x7fa8300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.027 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 0x7fa8300a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa838009c20 tx=0x7fa838005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:56.027 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 msgr2=0x7fa84819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa84000b970 tx=0x7fa84000be30 comp rx=0 tx=0).stop 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 shutdown_connections 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa848107430 0x7fa84819eff0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8300a2800 0x7fa8300a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa848104b00 0x7fa848197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 --2- 172.21.15.145:0/2352629721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8481021d0 0x7fa848197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.027+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 >> 172.21.15.145:0/2352629721 conn(0x7fa8480fbb40 msgr2=0x7fa848102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.028+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 shutdown_connections 2024-03-28T18:35:56.028 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.028+0000 7fa84d019640 1 -- 172.21.15.145:0/2352629721 wait complete. 2024-03-28T18:35:56.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 -- 172.21.15.145:0/365809272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 msgr2=0x7f4d841097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 --2- 172.21.15.145:0/365809272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d841097f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009ca0 tx=0x7f4d7c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 -- 172.21.15.145:0/365809272 shutdown_connections 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 --2- 172.21.15.145:0/365809272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d841097f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 --2- 172.21.15.145:0/365809272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84104a70 0x7f4d84106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 --2- 172.21.15.145:0/365809272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84102140 0x7f4d84104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 -- 172.21.15.145:0/365809272 >> 172.21.15.145:0/365809272 conn(0x7f4d840fbb30 msgr2=0x7f4d840fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 -- 172.21.15.145:0/365809272 shutdown_connections 2024-03-28T18:35:56.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.144+0000 7f4d88f52640 1 -- 172.21.15.145:0/365809272 wait complete. 2024-03-28T18:35:56.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 Processor -- start 2024-03-28T18:35:56.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 -- start start 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84102140 0x7f4d84197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84104a70 0x7f4d84197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8410d340 con 0x7f4d841073a0 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8410d1c0 con 0x7f4d84102140 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d88f52640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8410d4c0 con 0x7f4d84104a70 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49940/0 (socket says 172.21.15.145:49940) 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 -- 172.21.15.145:0/143008165 learned_addr learned my addr 172.21.15.145:0/143008165 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d81d9b640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84104a70 0x7f4d84197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d8259c640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84102140 0x7f4d84197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 -- 172.21.15.145:0/143008165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84104a70 msgr2=0x7f4d84197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84104a70 0x7f4d84197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 -- 172.21.15.145:0/143008165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84102140 msgr2=0x7f4d84197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84102140 0x7f4d84197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 -- 172.21.15.145:0/143008165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8419f5b0 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.145+0000 7f4d82d9d640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c037040 tx=0x7f4d7c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.146+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c041020 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.146+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8419f840 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.146+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d7c034d90 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.146+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c039040 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.146+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8419fd30 con 0x7f4d841073a0 2024-03-28T18:35:56.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.147+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d840fd620 con 0x7f4d841073a0 2024-03-28T18:35:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.148+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4d7c044020 con 0x7f4d841073a0 2024-03-28T18:35:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.148+0000 7f4d737fe640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.149+0000 7f4d8259c640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.149+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4d7c0f1040 con 0x7f4d841073a0 2024-03-28T18:35:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.149+0000 7f4d8259c640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009aa0 tx=0x7f4d74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.152+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d7c04b020 con 0x7f4d841073a0 2024-03-28T18:35:56.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:56 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T18:35:56.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:56 smithi145 ceph-mon[32151]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:35:56.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:56 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T18:35:56.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:56 smithi145 ceph-mon[32151]: pgmap v291: 129 pgs: 1 creating+activating, 7 creating+peering, 24 unknown, 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T18:35:56.366 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.366+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d8402d190 con 0x7f4d841073a0 2024-03-28T18:35:56.367 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.367+0000 7f4d737fe640 1 -- 172.21.15.145:0/143008165 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11348 (secure 0 0 0) 0x7f4d7c0bdea0 con 0x7f4d841073a0 2024-03-28T18:35:56.367 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:56.367 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:35:55.370846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24496,"qdb_cluster":[24496]},"id":1}]} 2024-03-28T18:35:56.368 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 17 2024-03-28T18:35:56.369 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 msgr2=0x7f4d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.369 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009aa0 tx=0x7f4d74005f90 comp rx=0 tx=0).stop 2024-03-28T18:35:56.369 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 msgr2=0x7f4d8419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c037040 tx=0x7f4d7c032f20 comp rx=0 tx=0).stop 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 shutdown_connections 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d841073a0 0x7f4d8419ef40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d6c0a2800 0x7f4d6c0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d84104a70 0x7f4d84197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 --2- 172.21.15.145:0/143008165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d84102140 0x7f4d84197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 >> 172.21.15.145:0/143008165 conn(0x7f4d840fbb30 msgr2=0x7f4d84102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 shutdown_connections 2024-03-28T18:35:56.370 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.369+0000 7f4d88f52640 1 -- 172.21.15.145:0/143008165 wait complete. 2024-03-28T18:35:56.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:35:56.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:56 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T18:35:56.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:56 smithi192 ceph-mon[36589]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:35:56.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:56 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T18:35:56.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:56 smithi192 ceph-mon[36589]: pgmap v291: 129 pgs: 1 creating+activating, 7 creating+peering, 24 unknown, 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T18:35:56.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:56 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3940708484' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T18:35:56.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:56 smithi164 ceph-mon[36248]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:35:56.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:56 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T18:35:56.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:56 smithi164 ceph-mon[36248]: pgmap v291: 129 pgs: 1 creating+activating, 7 creating+peering, 24 unknown, 97 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 -- 172.21.15.145:0/3729070535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 msgr2=0x7fb354101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 --2- 172.21.15.145:0/3729070535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb354101c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb348009900 tx=0x7fb34802bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 -- 172.21.15.145:0/3729070535 shutdown_connections 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 --2- 172.21.15.145:0/3729070535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb354102170 0x7fb354108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 --2- 172.21.15.145:0/3729070535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb354101c30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 --2- 172.21.15.145:0/3729070535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3541005c0 0x7fb3541009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 -- 172.21.15.145:0/3729070535 >> 172.21.15.145:0/3729070535 conn(0x7fb3540fbb30 msgr2=0x7fb3540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 -- 172.21.15.145:0/3729070535 shutdown_connections 2024-03-28T18:35:56.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.482+0000 7fb359937640 1 -- 172.21.15.145:0/3729070535 wait complete. 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 Processor -- start 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 -- start start 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3541005c0 0x7fb354069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb354102170 0x7fb35419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35410c240 con 0x7fb3541017b0 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35410c0c0 con 0x7fb3541005c0 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb359937640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35410c3c0 con 0x7fb354102170 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49962/0 (socket says 172.21.15.145:49962) 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 -- 172.21.15.145:0/3859978762 learned_addr learned my addr 172.21.15.145:0/3859978762 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3537fe640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb354102170 0x7fb35419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb352ffd640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3541005c0 0x7fb354069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 -- 172.21.15.145:0/3859978762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb354102170 msgr2=0x7fb35419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb354102170 0x7fb35419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 -- 172.21.15.145:0/3859978762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3541005c0 msgr2=0x7fb354069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3541005c0 0x7fb354069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.483+0000 7fb3527fc640 1 -- 172.21.15.145:0/3859978762 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35419f5e0 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb3527fc640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb348009230 tx=0x7fb348005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348041070 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35419f870 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3480044d0 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348034440 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.484+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35419fd60 con 0x7fb3541017b0 2024-03-28T18:35:56.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.485+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3541078e0 con 0x7fb3541017b0 2024-03-28T18:35:56.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.486+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb348044070 con 0x7fb3541017b0 2024-03-28T18:35:56.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.486+0000 7fb358935640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.486+0000 7fb352ffd640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.486+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb3480f0c00 con 0x7fb3541017b0 2024-03-28T18:35:56.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.487+0000 7fb352ffd640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb354101610 tx=0x7fb344009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.490+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3480bd410 con 0x7fb3541017b0 2024-03-28T18:35:56.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.665+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb35402d2c0 con 0x7fb3541017b0 2024-03-28T18:35:56.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.667+0000 7fb358935640 1 -- 172.21.15.145:0/3859978762 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fb3480c14b0 con 0x7fb3541017b0 2024-03-28T18:35:56.667 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:56.667 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 msgr2=0x7fb33c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb354101610 tx=0x7fb344009500 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 msgr2=0x7fb35419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb348009230 tx=0x7fb348005d50 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 shutdown_connections 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb33c0a2800 0x7fb33c0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb354102170 0x7fb35419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3541017b0 0x7fb35419c6e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 --2- 172.21.15.145:0/3859978762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3541005c0 0x7fb354069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 >> 172.21.15.145:0/3859978762 conn(0x7fb3540fbb30 msgr2=0x7fb354104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 shutdown_connections 2024-03-28T18:35:56.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.669+0000 7fb359937640 1 -- 172.21.15.145:0/3859978762 wait complete. 2024-03-28T18:35:56.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:35:56.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.781+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3666916122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1017b0 msgr2=0x7fe79c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.781+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3666916122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1017b0 0x7fe79c101c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009900 tx=0x7fe78c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.781+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3666916122 shutdown_connections 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3666916122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c102170 0x7fe79c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3666916122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1017b0 0x7fe79c101c30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3666916122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c1005c0 0x7fe79c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3666916122 >> 172.21.15.145:0/3666916122 conn(0x7fe79c0fbb30 msgr2=0x7fe79c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3666916122 shutdown_connections 2024-03-28T18:35:56.782 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3666916122 wait complete. 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 Processor -- start 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- start start 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c1017b0 0x7fe79c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 0x7fe79c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.784 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c10c280 con 0x7fe79c1005c0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c10c100 con 0x7fe79c102170 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe7a0af8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c10c400 con 0x7fe79c1017b0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.782+0000 7fe79a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49984/0 (socket says 172.21.15.145:49984) 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 -- 172.21.15.145:0/3348261603 learned_addr learned my addr 172.21.15.145:0/3348261603 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe799d9b640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c1017b0 0x7fe79c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79ad9d640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 0x7fe79c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 -- 172.21.15.145:0/3348261603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c1017b0 msgr2=0x7fe79c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c1017b0 0x7fe79c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 -- 172.21.15.145:0/3348261603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 msgr2=0x7fe79c1030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 0x7fe79c1030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 -- 172.21.15.145:0/3348261603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c103960 con 0x7fe79c1005c0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79ad9d640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 0x7fe79c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe79a59c640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784004220 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7840043c0 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c1a1b40 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78400f450 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.783+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c1a2060 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.784+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79c068490 con 0x7fe79c1005c0 2024-03-28T18:35:56.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.785+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe7840079b0 con 0x7fe79c1005c0 2024-03-28T18:35:56.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.786+0000 7fe7937fe640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 0x7fe7880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:56.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.786+0000 7fe799d9b640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 0x7fe7880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:56.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.786+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe7840cea60 con 0x7fe79c1005c0 2024-03-28T18:35:56.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.786+0000 7fe799d9b640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 0x7fe7880a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe78c02c240 tx=0x7fe78c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:56.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.790+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe784014030 con 0x7fe79c1005c0 2024-03-28T18:35:56.968 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.968+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe79c1a29a0 con 0x7fe79c1005c0 2024-03-28T18:35:56.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.970+0000 7fe7937fe640 1 -- 172.21.15.145:0/3348261603 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fe78409b8c0 con 0x7fe79c1005c0 2024-03-28T18:35:56.970 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:56.970 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 msgr2=0x7fe7880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 0x7fe7880a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe78c02c240 tx=0x7fe78c02df90 comp rx=0 tx=0).stop 2024-03-28T18:35:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 msgr2=0x7fe79c104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:56.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).stop 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 shutdown_connections 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe79c102170 0x7fe79c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe7880a2800 0x7fe7880a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe79c1017b0 0x7fe79c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 --2- 172.21.15.145:0/3348261603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe79c1005c0 0x7fe79c104550 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 >> 172.21.15.145:0/3348261603 conn(0x7fe79c0fbb30 msgr2=0x7fe79c0fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 shutdown_connections 2024-03-28T18:35:56.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:56.972+0000 7fe7a0af8640 1 -- 172.21.15.145:0/3348261603 wait complete. 2024-03-28T18:35:56.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 -- 172.21.15.145:0/2613777341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c103280 msgr2=0x7f4d0c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/2613777341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c103280 0x7f4d0c109b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009ca0 tx=0x7f4d0402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 -- 172.21.15.145:0/2613777341 shutdown_connections 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/2613777341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c103280 0x7f4d0c109b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/2613777341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c1028c0 0x7f4d0c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/2613777341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 -- 172.21.15.145:0/2613777341 >> 172.21.15.145:0/2613777341 conn(0x7f4d0c0fcc80 msgr2=0x7f4d0c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 -- 172.21.15.145:0/2613777341 shutdown_connections 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.099+0000 7f4d10c0b640 1 -- 172.21.15.145:0/2613777341 wait complete. 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 Processor -- start 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 -- start start 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d0c1028c0 0x7f4d0c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c103280 0x7f4d0c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c10d330 con 0x7f4d0c1016d0 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c10d1b0 con 0x7f4d0c1028c0 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d10c0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c10d4b0 con 0x7f4d0c103280 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50010/0 (socket says 172.21.15.145:50010) 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 -- 172.21.15.145:0/1868226914 learned_addr learned my addr 172.21.15.145:0/1868226914 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0ad9d640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c103280 0x7f4d0c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 -- 172.21.15.145:0/1868226914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c103280 msgr2=0x7f4d0c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c103280 0x7f4d0c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 -- 172.21.15.145:0/1868226914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d0c1028c0 msgr2=0x7f4d0c197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d0c1028c0 0x7f4d0c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.100+0000 7f4d0a59c640 1 -- 172.21.15.145:0/1868226914 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0c19f560 con 0x7f4d0c1016d0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d0a59c640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00d950 tx=0x7f4cfc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc017020 con 0x7f4d0c1016d0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cfc009c30 con 0x7f4d0c1016d0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0c19f850 con 0x7f4d0c1016d0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00f740 con 0x7f4d0c1016d0 2024-03-28T18:35:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.101+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c19fda0 con 0x7f4d0c1016d0 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.102+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c1089f0 con 0x7f4d0c1016d0 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.103+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4cfc01a020 con 0x7f4d0c1016d0 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.104+0000 7f4d037fe640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 0x7f4cf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.104+0000 7f4d03fff640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 0x7f4cf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.104+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4cfc0ce6f0 con 0x7f4d0c1016d0 2024-03-28T18:35:57.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.104+0000 7f4d03fff640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 0x7f4cf80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c198e50 tx=0x7f4cf4009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.108+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cfc021070 con 0x7f4d0c1016d0 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/143008165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3859978762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:57 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3348261603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.320+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d0c02d070 con 0x7f4d0c1016d0 2024-03-28T18:35:57.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.321+0000 7f4d037fe640 1 -- 172.21.15.145:0/1868226914 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11348 (secure 0 0 0) 0x7f4cfc009de0 con 0x7f4d0c1016d0 2024-03-28T18:35:57.321 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:57.321 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24491,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:35:55.370846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24496,"qdb_cluster":[24496]},"id":1}]} 2024-03-28T18:35:57.321 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 17 2024-03-28T18:35:57.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 msgr2=0x7f4cf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 0x7f4cf80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c198e50 tx=0x7f4cf4009590 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 msgr2=0x7f4d0c197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00d950 tx=0x7f4cfc00de10 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 shutdown_connections 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4cf80a2800 0x7f4cf80a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d0c103280 0x7f4d0c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d0c1028c0 0x7f4d0c197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 --2- 172.21.15.145:0/1868226914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d0c1016d0 0x7f4d0c197940 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 >> 172.21.15.145:0/1868226914 conn(0x7f4d0c0fcc80 msgr2=0x7f4d0c105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.323+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 shutdown_connections 2024-03-28T18:35:57.324 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.324+0000 7f4d10c0b640 1 -- 172.21.15.145:0/1868226914 wait complete. 2024-03-28T18:35:57.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:35:57.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:57.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:35:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T18:35:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/143008165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3859978762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:57 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3348261603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 -- 172.21.15.145:0/3142093366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 msgr2=0x7fd4f00ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/3142093366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f00ffbc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009900 tx=0x7fd4e402bc30 comp rx=0 tx=0).stop 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 -- 172.21.15.145:0/3142093366 shutdown_connections 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/3142093366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 0x7fd4f0109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/3142093366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f00ffbc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.439 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/3142093366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 -- 172.21.15.145:0/3142093366 >> 172.21.15.145:0/3142093366 conn(0x7fd4f00fb2c0 msgr2=0x7fd4f00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 -- 172.21.15.145:0/3142093366 shutdown_connections 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 -- 172.21.15.145:0/3142093366 wait complete. 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.437+0000 7fd4f63e6640 1 Processor -- start 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 -- start start 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 0x7fd4f019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f010d360 con 0x7fd4f00ff740 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f010d1e0 con 0x7fd4f0102cd0 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f63e6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f010d4e0 con 0x7fd4f0100100 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50038/0 (socket says 172.21.15.145:50038) 2024-03-28T18:35:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 -- 172.21.15.145:0/4115632005 learned_addr learned my addr 172.21.15.145:0/4115632005 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4ef7fe640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 0x7fd4f019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f4983640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 -- 172.21.15.145:0/4115632005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 msgr2=0x7fd4f019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 0x7fd4f019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 -- 172.21.15.145:0/4115632005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 msgr2=0x7fd4f01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 -- 172.21.15.145:0/4115632005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f01a1810 con 0x7fd4f00ff740 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4f4983640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4effff640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000d950 tx=0x7fd4e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.438+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0017020 con 0x7fd4f00ff740 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.439+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4e0009c30 con 0x7fd4f00ff740 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.439+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f01a1b00 con 0x7fd4f00ff740 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.439+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000f740 con 0x7fd4f00ff740 2024-03-28T18:35:57.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.439+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f01a2020 con 0x7fd4f00ff740 2024-03-28T18:35:57.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.440+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f01a21f0 con 0x7fd4f00ff740 2024-03-28T18:35:57.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.441+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd4e001a020 con 0x7fd4f00ff740 2024-03-28T18:35:57.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.441+0000 7fd4ed7fa640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 0x7fd4d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.442+0000 7fd4ef7fe640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 0x7fd4d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.442+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd4e00ce690 con 0x7fd4f00ff740 2024-03-28T18:35:57.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.442+0000 7fd4ef7fe640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 0x7fd4d80a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4e402c7b0 tx=0x7fd4e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.446+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4e0021070 con 0x7fd4f00ff740 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2352629721' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/143008165' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3859978762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:57 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3348261603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:57.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.622+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4f01a23c0 con 0x7fd4f00ff740 2024-03-28T18:35:57.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.624+0000 7fd4ed7fa640 1 -- 172.21.15.145:0/4115632005 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fd4e0009de0 con 0x7fd4f00ff740 2024-03-28T18:35:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:35:57.624 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.625+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 msgr2=0x7fd4d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.625+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 0x7fd4d80a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4e402c7b0 tx=0x7fd4e4005fd0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 msgr2=0x7fd4f0199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000d950 tx=0x7fd4e000de10 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 shutdown_connections 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4f0102cd0 0x7fd4f01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4d80a2800 0x7fd4d80a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4f0100100 0x7fd4f019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 --2- 172.21.15.145:0/4115632005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4f00ff740 0x7fd4f0199bf0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 >> 172.21.15.145:0/4115632005 conn(0x7fd4f00fb2c0 msgr2=0x7fd4f0100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 shutdown_connections 2024-03-28T18:35:57.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.626+0000 7fd4f63e6640 1 -- 172.21.15.145:0/4115632005 wait complete. 2024-03-28T18:35:57.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T18:35:50.266038+0000', 'modified': '2024-03-28T18:35:55.370846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24496}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.192:6839/2516430938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 2516430938}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 2516430938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24496, 'qdb_cluster': [24496]} 2024-03-28T18:35:57.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T18:35:57.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:35:57.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.739+0000 7f7232cf1640 1 -- 172.21.15.145:0/2800659752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c103260 msgr2=0x7f722c109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.739+0000 7f7232cf1640 1 --2- 172.21.15.145:0/2800659752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c103260 0x7f722c109af0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7224009ca0 tx=0x7f722402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- 172.21.15.145:0/2800659752 shutdown_connections 2024-03-28T18:35:57.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- 172.21.15.145:0/2800659752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c103260 0x7f722c109af0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- 172.21.15.145:0/2800659752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f722c1028a0 0x7f722c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- 172.21.15.145:0/2800659752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 0x7f722c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- 172.21.15.145:0/2800659752 >> 172.21.15.145:0/2800659752 conn(0x7f722c0fcc40 msgr2=0x7f722c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- 172.21.15.145:0/2800659752 shutdown_connections 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- 172.21.15.145:0/2800659752 wait complete. 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 Processor -- start 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- start start 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 0x7f722c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f722c103260 0x7f722c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c10d310 con 0x7f722c1028a0 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c10d190 con 0x7f722c103260 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7232cf1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c10d490 con 0x7f722c1016b0 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f722bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f722bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50044/0 (socket says 172.21.15.145:50044) 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f722bfff640 1 -- 172.21.15.145:0/1530229490 learned_addr learned my addr 172.21.15.145:0/1530229490 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.740+0000 7f7230a8d640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 0x7f722c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 -- 172.21.15.145:0/1530229490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 msgr2=0x7f722c197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 0x7f722c197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 -- 172.21.15.145:0/1530229490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f722c103260 msgr2=0x7f722c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f722c103260 0x7f722c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 -- 172.21.15.145:0/1530229490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c19f570 con 0x7f722c1028a0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f722bfff640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9f0 tx=0x7f721800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f7229ffb640 1 -- 172.21.15.145:0/1530229490 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721800cd80 con 0x7f722c1028a0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c19f860 con 0x7f722c1028a0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f7229ffb640 1 -- 172.21.15.145:0/1530229490 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7218010910 con 0x7f722c1028a0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f7229ffb640 1 -- 172.21.15.145:0/1530229490 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218018990 con 0x7f722c1028a0 2024-03-28T18:35:57.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.741+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c19fdb0 con 0x7f722c1028a0 2024-03-28T18:35:57.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.743+0000 7f7229ffb640 1 -- 172.21.15.145:0/1530229490 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7218010ac0 con 0x7f722c1028a0 2024-03-28T18:35:57.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.743+0000 7f72359bc740 1 Processor -- start 2024-03-28T18:35:57.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 -- start start 2024-03-28T18:35:57.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960ebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960ee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f39604bc20 con 0x55f3960e6740 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f7229ffb640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 0x7f721c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f395ff4a40 con 0x55f3960e7130 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f72359bc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f396047cc0 con 0x55f3960ec2a0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960e6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50048/0 (socket says 172.21.15.145:50048) 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f723128e640 1 -- 172.21.15.145:0/3670936784 learned_addr learned my addr 172.21.15.145:0/3670936784 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f723128e640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 0x7f721c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f7230a8d640 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960ebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f7229ffb640 1 -- 172.21.15.145:0/1530229490 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f72180cf0a0 con 0x7f722c1028a0 2024-03-28T18:35:57.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 -- 172.21.15.145:0/3670936784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 msgr2=0x55f3960ebc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960ebc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 -- 172.21.15.145:0/3670936784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 msgr2=0x55f3960e6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960e6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 -- 172.21.15.145:0/3670936784 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f396016f90 con 0x55f3960ec2a0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f7230a8d640 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960ebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f723128e640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 0x7f721c0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55f3960e7990 tx=0x7f7224032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f722bfff640 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960ee730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f721806e5c0 tx=0x7f721806d540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f1b419f7078055a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.744+0000 7f7216ffd640 1 -- 172.21.15.145:0/3670936784 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218026040 con 0x55f3960ec2a0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f7216ffd640 1 -- 172.21.15.145:0/3670936784 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f721809bf00 con 0x55f3960ec2a0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f7216ffd640 1 -- 172.21.15.145:0/3670936784 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72180860d0 con 0x55f3960ec2a0 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 -- 172.21.15.145:0/3670936784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 msgr2=0x55f3960ee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960ee730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f721806e5c0 tx=0x7f721806d540 comp rx=0 tx=0).stop 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 -- 172.21.15.145:0/3670936784 shutdown_connections 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960ee730 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960ebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 --2- 172.21.15.145:0/3670936784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 -- 172.21.15.145:0/3670936784 >> 172.21.15.145:0/3670936784 conn(0x55f3960aea60 msgr2=0x55f3960e20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 -- 172.21.15.145:0/3670936784 shutdown_connections 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.745+0000 7f72359bc740 1 -- 172.21.15.145:0/3670936784 wait complete. 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 -- start start 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 Processor -- start 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960f8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960ef250 con 0x55f3960e6740 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f396007e90 con 0x55f3960e7130 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960ef3d0 con 0x55f3960ec2a0 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f72359bc740 10 client.0 _ll_register_callbacks cb 0x55f395d5fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50052/0 (socket says 172.21.15.145:50052) 2024-03-28T18:35:57.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 -- 172.21.15.145:0/1880907644 learned_addr learned my addr 172.21.15.145:0/1880907644 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f723128e640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f722bfff640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 -- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 msgr2=0x55f3960f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 -- 172.21.15.145:0/1880907644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 msgr2=0x55f3960f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960f11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f7230a8d640 1 -- 172.21.15.145:0/1880907644 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3960149d0 con 0x55f3960e6740 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f723128e640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.746+0000 7f722bfff640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f7230a8d640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7220012700 tx=0x7f7220012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f0a9070612d0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220004030 con 0x55f3960e6740 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f721809bf00 con 0x55f3960e6740 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f72200041d0 con 0x55f3960e6740 2024-03-28T18:35:57.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7220004410 con 0x55f3960e6740 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 10 client.14757 fetch_fsmap learned FSMap version 17 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55f395d602d0 con 0x55f3960e6740 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f721809bf00 con 0x55f3960e6740 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f7220003b40 con 0x55f3960e6740 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 10 client.14757 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 10 client.14757 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 20 client.14757 populate_metadata read hostname 'smithi145' 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3960fd5d0 0x55f3960fc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 4 client.14757 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f396059160 con 0x55f3960fd5d0 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.747+0000 7f722bfff640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3960fd5d0 0x55f3960fc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f720bfff640 10 client.14757 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f722bfff640 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3960fd5d0 0x55f3960fc620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71f1e834b9c1fb72 server_cookie=84365a9c881699e6 in_seq=0 out_seq=0 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f720bfff640 1 -- 172.21.15.145:0/1880907644 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f396059160 con 0x55f3960fd5d0 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f720bfff640 10 client.14757 handle_command_reply: tid=0 2024-03-28T18:35:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f72359bc740 1 client.14757 shutdown 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 msgr2=0x55f3960f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.748+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7220012700 tx=0x7f7220012bc0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 shutdown_connections 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3960fd5d0 0x55f3960fc620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960ec2a0 0x55f3960f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f720bfff640 0 client.14757 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e7130 0x55f3960f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 --2- 172.21.15.145:0/1880907644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960e6740 0x55f3960f0cb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 >> 172.21.15.145:0/1880907644 conn(0x55f3960aea60 msgr2=0x55f3960e12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 shutdown_connections 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 1 -- 172.21.15.145:0/1880907644 wait complete. 2024-03-28T18:35:57.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.749+0000 7f72359bc740 20 client.14757 trim_cache size 0 max 16384 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 Processor -- start 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 -- start start 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3960dd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f3960e0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fcdf0 con 0x55f3960ec2a0 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fd450 con 0x55f3960e6740 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f72359bc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fdab0 con 0x55f3960e7130 2024-03-28T18:35:57.770 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.765+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50056/0 (socket says 172.21.15.145:50056) 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 -- 172.21.15.145:0/2355585314 learned_addr learned my addr 172.21.15.145:0/2355585314 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f722bfff640 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f3960e0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f7230a8d640 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3960dd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 -- 172.21.15.145:0/2355585314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 msgr2=0x55f3960e0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f3960e0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 -- 172.21.15.145:0/2355585314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 msgr2=0x55f3960dd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3960dd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 -- 172.21.15.145:0/2355585314 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f395e2a6b0 con 0x55f3960ec2a0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f723128e640 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7224009d90 tx=0x7f722403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3977bdc4e2f1d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f72167fc640 1 -- 172.21.15.145:0/2355585314 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224048050 con 0x55f3960ec2a0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f72167fc640 1 -- 172.21.15.145:0/2355585314 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7224052070 con 0x55f3960ec2a0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.766+0000 7f72167fc640 1 -- 172.21.15.145:0/2355585314 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224039040 con 0x55f3960ec2a0 2024-03-28T18:35:57.771 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- 172.21.15.145:0/2355585314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 msgr2=0x55f3960e0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7224009d90 tx=0x7f722403f000 comp rx=0 tx=0).stop 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- 172.21.15.145:0/2355585314 shutdown_connections 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f3960e0fa0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f3960e0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- 172.21.15.145:0/2355585314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3960dd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- 172.21.15.145:0/2355585314 >> 172.21.15.145:0/2355585314 conn(0x55f3960aea60 msgr2=0x55f3960e7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- 172.21.15.145:0/2355585314 shutdown_connections 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- 172.21.15.145:0/2355585314 wait complete. 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- start start 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 Processor -- start 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3961544d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fe770 con 0x55f3960ec2a0 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fe110 con 0x55f3960e6740 2024-03-28T18:35:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3960fedd0 con 0x55f3960e7130 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f72359bc740 10 client.0 _ll_register_callbacks cb 0x7f7220009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.767+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f722bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50064/0 (socket says 172.21.15.145:50064) 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f722bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:58086/0 (socket says 172.21.15.145:58086) 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 -- 172.21.15.145:0/3503937129 learned_addr learned my addr 172.21.15.145:0/3503937129 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f7230a8d640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3961544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 -- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 msgr2=0x55f396154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 -- 172.21.15.145:0/3503937129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 msgr2=0x55f3961544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3961544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 -- 172.21.15.145:0/3503937129 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f39615bfd0 con 0x55f3960ec2a0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f722bfff640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f723128e640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72240324c0 tx=0x7f7224003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f5225a5773a47d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:57.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722405a020 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7224004360 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7224052070 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7224048020 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f72359bc740 10 client.14763 fetch_fsmap learned FSMap version 17 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55f39615d2d0 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7224052070 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.768+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f722403f070 con 0x55f3960ec2a0 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 10 client.14763 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 10 client.14763 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 20 client.14763 populate_metadata read hostname 'smithi145' 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3961615c0 0x55f39615d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 4 client.14763 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:35:57.774 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f3960534b0 con 0x55f3961615c0 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f7230a8d640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3961615c0 0x55f39615d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f720bfff640 10 client.14763 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f7230a8d640 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3961615c0 0x55f39615d4a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebda0b8cc0f7140 server_cookie=833ccdd30ce8ca27 in_seq=0 out_seq=0 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f720bfff640 1 -- 172.21.15.145:0/3503937129 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f3960534b0 con 0x55f3961615c0 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f720bfff640 10 client.14763 handle_command_reply: tid=0 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 client.14763 shutdown 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 msgr2=0x55f39615b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72240324c0 tx=0x7f7224003f60 comp rx=0 tx=0).stop 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 shutdown_connections 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f3961615c0 0x55f39615d4a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f3960ec2a0 0x55f39615b9e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f720bfff640 0 client.14763 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f3960e7130 0x55f396154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 --2- 172.21.15.145:0/3503937129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3960e6740 0x55f3961544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 >> 172.21.15.145:0/3503937129 conn(0x55f3960aea60 msgr2=0x55f3960e52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 shutdown_connections 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.769+0000 7f72359bc740 1 -- 172.21.15.145:0/3503937129 wait complete. 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f72359bc740 20 client.14763 trim_cache size 0 max 16384 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 0, 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24496, 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 6.8512637600000001, 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 17, 2024-03-28T18:35:57.776 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 91, 2024-03-28T18:35:57.777 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:35:57.777 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 22.254436213999998 2024-03-28T18:35:57.777 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:35:57.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 msgr2=0x7f721c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 0x7f721c0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55f3960e7990 tx=0x7f7224032590 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 msgr2=0x7f722c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9f0 tx=0x7f721800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 shutdown_connections 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f722c103260 0x7f722c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f722c1028a0 0x7f722c197e90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f721c0a2800 0x7f721c0a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 --2- 172.21.15.145:0/1530229490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f722c1016b0 0x7f722c197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 >> 172.21.15.145:0/1530229490 conn(0x7f722c0fcc40 msgr2=0x7f722c105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 shutdown_connections 2024-03-28T18:35:57.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:57.770+0000 7f7232cf1640 1 -- 172.21.15.145:0/1530229490 wait complete. 2024-03-28T18:35:57.781 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:35:57.781 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:35:57.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T18:35:58.017 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:35:58.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:58 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1868226914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:58.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:58 smithi145 ceph-mon[32151]: pgmap v293: 129 pgs: 1 creating+activating, 7 creating+peering, 21 unknown, 100 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 5 op/s 2024-03-28T18:35:58.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:58 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4115632005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:58.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:58 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1868226914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:58.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:58 smithi192 ceph-mon[36589]: pgmap v293: 129 pgs: 1 creating+activating, 7 creating+peering, 21 unknown, 100 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 5 op/s 2024-03-28T18:35:58.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:58 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4115632005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:58.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:58 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1868226914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:35:58.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:58 smithi164 ceph-mon[36248]: pgmap v293: 129 pgs: 1 creating+activating, 7 creating+peering, 21 unknown, 100 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 5 op/s 2024-03-28T18:35:58.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:58 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4115632005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:35:59.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:59 smithi145 ceph-mon[32151]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:59.376 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:35:59 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:35:59.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:59 smithi192 ceph-mon[36589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:59.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:35:59 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:35:59.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:59 smithi164 ceph-mon[36248]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T18:35:59.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:35:59 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 -- 172.21.15.145:0/3415280285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a541021e0 msgr2=0x7f1a54102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 --2- 172.21.15.145:0/3415280285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a541021e0 0x7f1a54102660 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1a480099c0 tx=0x7f1a48009cd0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 -- 172.21.15.145:0/3415280285 shutdown_connections 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 --2- 172.21.15.145:0/3415280285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a54102ba0 0x7f1a5410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 --2- 172.21.15.145:0/3415280285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541081c0 0x7f1a541085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 --2- 172.21.15.145:0/3415280285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a541021e0 0x7f1a54102660 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 -- 172.21.15.145:0/3415280285 >> 172.21.15.145:0/3415280285 conn(0x7f1a540792c0 msgr2=0x7f1a540796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 -- 172.21.15.145:0/3415280285 shutdown_connections 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.786+0000 7f1a5a526700 1 -- 172.21.15.145:0/3415280285 wait complete. 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 Processor -- start 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 -- start start 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 0x7f1a5419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 0x7f1a54198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a541124f0 con 0x7f1a54102ba0 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54112370 con 0x7f1a541021e0 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a5a526700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54112670 con 0x7f1a541081c0 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a537fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:50068/0 (socket says 172.21.15.145:50068) 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a537fe700 1 -- 172.21.15.145:0/2362282672 learned_addr learned my addr 172.21.15.145:0/2362282672 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:35:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a58ac3700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 0x7f1a54198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.787+0000 7f1a53fff700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 0x7f1a5419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 -- 172.21.15.145:0/2362282672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 msgr2=0x7f1a54198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 0x7f1a54198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 -- 172.21.15.145:0/2362282672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 msgr2=0x7f1a5419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 0x7f1a5419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 -- 172.21.15.145:0/2362282672 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54199010 con 0x7f1a54102ba0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a53fff700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 0x7f1a5419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a58ac3700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 0x7f1a54198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a537fe700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a480059d0 tx=0x7f1a4800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48027070 con 0x7f1a54102ba0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a541992a0 con 0x7f1a54102ba0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a48003680 con 0x7f1a54102ba0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48017650 con 0x7f1a54102ba0 2024-03-28T18:35:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.788+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a541a54c0 con 0x7f1a54102ba0 2024-03-28T18:35:59.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.789+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a54103870 con 0x7f1a54102ba0 2024-03-28T18:35:59.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.790+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1a4802a070 con 0x7f1a54102ba0 2024-03-28T18:35:59.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.791+0000 7f1a517fa700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 0x7f1a3c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:59.792 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.791+0000 7f1a53fff700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 0x7f1a3c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:59.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.791+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1a480cfb30 con 0x7f1a54102ba0 2024-03-28T18:35:59.793 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.791+0000 7f1a53fff700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 0x7f1a3c0a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a440099d0 tx=0x7f1a44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:59.796 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:35:59.795+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a4805e740 con 0x7f1a54102ba0 2024-03-28T18:36:00.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.034+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1a54199d10 con 0x7f1a54102ba0 2024-03-28T18:36:00.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.035+0000 7f1a517fa700 1 -- 172.21.15.145:0/2362282672 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1a48013070 con 0x7f1a54102ba0 2024-03-28T18:36:00.036 INFO:teuthology.orchestra.run.smithi145.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 msgr2=0x7f1a3c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 0x7f1a3c0a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a440099d0 tx=0x7f1a44008040 comp rx=0 tx=0).stop 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 msgr2=0x7f1a5419eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a480059d0 tx=0x7f1a4800fa00 comp rx=0 tx=0).stop 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 shutdown_connections 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a3c0a2890 0x7f1a3c0a4d50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a541081c0 0x7f1a54198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:00.038 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a541021e0 0x7f1a5419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:00.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 --2- 172.21.15.145:0/2362282672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a54102ba0 0x7f1a5419eba0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:00.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 >> 172.21.15.145:0/2362282672 conn(0x7f1a540792c0 msgr2=0x7f1a54100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:00.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 shutdown_connections 2024-03-28T18:36:00.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:00.037+0000 7f1a5a526700 1 -- 172.21.15.145:0/2362282672 wait complete. 2024-03-28T18:36:00.357 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:00 smithi145 ceph-mon[32151]: pgmap v294: 129 pgs: 1 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-28T18:36:00.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:00 smithi192 ceph-mon[36589]: pgmap v294: 129 pgs: 1 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-28T18:36:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:00 smithi164 ceph-mon[36248]: pgmap v294: 129 pgs: 1 creating+activating, 7 creating+peering, 121 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi145.front.sepia.ceph.com 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:36:00.696 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T18:36:00.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:00.697 DEBUG:teuthology.orchestra.run.smithi145:> ip addr 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:66 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f0 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:67 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f1 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:36:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:90 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:91 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.21.15.145/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft 42239sec preferred_lft 42239sec 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-03-28T18:36:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:36:00.714 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:36:00.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr flush dev ceph-brx 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set ceph-brx up 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:36:00.715 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:36:00.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:00.878 DEBUG:teuthology.orchestra.run.smithi145:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:36:00.905 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-03-28T18:36:00.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:00.906 DEBUG:teuthology.orchestra.run.smithi145:> route 2024-03-28T18:36:00.969 INFO:teuthology.orchestra.run.smithi145.stdout:Kernel IP routing table 2024-03-28T18:36:00.969 INFO:teuthology.orchestra.run.smithi145.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:36:00.969 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T18:36:00.969 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T18:36:00.969 INFO:teuthology.orchestra.run.smithi145.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:36:00.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T18:36:00.970 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:36:01.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.247 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-28T18:36:01.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.262 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list-id 2024-03-28T18:36:01.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.316 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:36:01.316 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:36:01.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:36:01.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T18:36:01.317 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:36:01.381 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:01 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2362282672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:36:01.399 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T18:36:01.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T18:36:01.400 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:36:01.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:01 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2362282672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:36:01.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:01 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2362282672' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:36:01.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.540 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:36:01.540 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:36:01.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set brx.0 up 2024-03-28T18:36:01.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:36:01.540 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:36:01.581 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T18:36:01.581 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:36:01.581 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:01.636 INFO:teuthology.orchestra.run.smithi145.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T18:36:01.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:36:01.636 DEBUG:teuthology.orchestra.run.smithi145:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:01.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-03-28T18:36:01.758 DEBUG:teuthology.orchestra.run.smithi145:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:36:01.813 INFO:teuthology.orchestra.run.smithi145.stdout:/proc 2024-03-28T18:36:01.813 INFO:teuthology.orchestra.run.smithi145.stdout:/sys 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/dev 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/security 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/shm 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/pts 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/run 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/cgroup 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/pstore 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/bpf 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/ 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/selinux 2024-03-28T18:36:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/hugepages 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/mqueue 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/debug 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/tracing 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/fuse/connections 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/config 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/run/user/1000 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/ceph 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/b6fef1b3c7d369a6ce6406fc8506970a1e198cd5cecb81045026e4a68de28a30/merged 2024-03-28T18:36:01.815 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/a6510e3e7b4c9767dcdade300ef5a25dc0a1ccd76f346b1abc64a89c43627369/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/run/user/0 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/09e3aa12846c7bc7c65a1528cabb40e3253127f7695544263e4e296ac4764f36/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/59028a826f9d1339859d94dce99519441504ae91daead83929c33e62a12a7f27/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/c3ebcb63357aff895fe18749e2f1cc5f2361218c21f36daf8f7516ab7d75c6d2/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/a6b1800071d250d283ed6207e6ee2cdb809833246a070c7ded3b8f5a641d8933/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/1772f2098a64c82a302978db39de1ca8092cb851c7ce7d5ed6877853f07f698b/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/c5e44459b75d232ff0d016a245dbbb0925bdb88c7ca46c45884025d6b2574ae9/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/6fc1b119a39ba32a5a5079103e9fe4cb29e8c313198a6240c095a43837c4fb31/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/dc314c9cd1d725f8256a92c45b608a38a8e63e7c9312566a8b1ff3ac064eefd8/merged 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:36:01.816 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:36:01.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:01.817 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-03-28T18:36:01.872 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T18:36:01.872 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T18:36:01.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-03-28T18:36:01.941 DEBUG:teuthology.orchestra.run.smithi145:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:36:01.998 INFO:teuthology.orchestra.run.smithi145.stdout:/proc 2024-03-28T18:36:01.998 INFO:teuthology.orchestra.run.smithi145.stdout:/sys 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/dev 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/security 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/shm 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/pts 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/run 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/cgroup 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/pstore 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/bpf 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/ 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/selinux 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/hugepages 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/dev/mqueue 2024-03-28T18:36:01.999 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/debug 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/tracing 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/fs/fuse/connections 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/sys/kernel/config 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/run/user/1000 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/ceph 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/b6fef1b3c7d369a6ce6406fc8506970a1e198cd5cecb81045026e4a68de28a30/merged 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/a6510e3e7b4c9767dcdade300ef5a25dc0a1ccd76f346b1abc64a89c43627369/merged 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/run/user/0 2024-03-28T18:36:02.000 INFO:teuthology.orchestra.run.smithi145.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/09e3aa12846c7bc7c65a1528cabb40e3253127f7695544263e4e296ac4764f36/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/59028a826f9d1339859d94dce99519441504ae91daead83929c33e62a12a7f27/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/c3ebcb63357aff895fe18749e2f1cc5f2361218c21f36daf8f7516ab7d75c6d2/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/a6b1800071d250d283ed6207e6ee2cdb809833246a070c7ded3b8f5a641d8933/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/1772f2098a64c82a302978db39de1ca8092cb851c7ce7d5ed6877853f07f698b/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/c5e44459b75d232ff0d016a245dbbb0925bdb88c7ca46c45884025d6b2574ae9/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/6fc1b119a39ba32a5a5079103e9fe4cb29e8c313198a6240c095a43837c4fb31/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/var/lib/containers/storage/overlay/dc314c9cd1d725f8256a92c45b608a38a8e63e7c9312566a8b1ff3ac064eefd8/merged 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run.smithi145.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:36:02.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:02.001 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-03-28T18:36:02.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi145.stderr:2024-03-28T18:36:02.006+0000 7fd7a9a41480 -1 init, newargv = 0x560b78c6bcc0 newargc=15 2024-03-28T18:36:02.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi145.stderr:ceph-fuse[73276]: starting ceph client 2024-03-28T18:36:02.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi145.stderr:ceph-fuse[73276]: starting fuse 2024-03-28T18:36:02.022 INFO:teuthology.orchestra.run.smithi145.stdout:119 2024-03-28T18:36:02.022 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo stdin-killer -- python3 -c ' 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> import glob 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> import re 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> import os 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> import subprocess 2024-03-28T18:36:02.022 DEBUG:teuthology.orchestra.run.smithi145:> 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> def _find_admin_socket(client_name): 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> files = glob.glob(asok_path) 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> # Given a non-glob path, it better be there 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> if "*" not in asok_path: 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> assert(len(files) == 1) 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> return files[0] 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> for f in files: 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> contents = proc_f.read() 2024-03-28T18:36:02.023 DEBUG:teuthology.orchestra.run.smithi145:> if mountpoint in contents: 2024-03-28T18:36:02.024 DEBUG:teuthology.orchestra.run.smithi145:> return f 2024-03-28T18:36:02.024 DEBUG:teuthology.orchestra.run.smithi145:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T18:36:02.024 DEBUG:teuthology.orchestra.run.smithi145:> 2024-03-28T18:36:02.024 DEBUG:teuthology.orchestra.run.smithi145:> print(_find_admin_socket("client.admin")) 2024-03-28T18:36:02.024 DEBUG:teuthology.orchestra.run.smithi145:> ' 2024-03-28T18:36:02.144 INFO:teuthology.orchestra.run.smithi145.stdout:/var/run/ceph/ceph-client.admin.73276.asok 2024-03-28T18:36:02.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T18:36:02.154 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73276.asok 2024-03-28T18:36:02.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:02.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73276.asok status 2024-03-28T18:36:02.292 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "metadata": { 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "entity_id": "admin", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": "smithi145", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "pid": "73276", 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "root": "/" 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_count": 0, 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_pinned_count": 0, 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 14775, 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "inst": { 2024-03-28T18:36:02.293 INFO:teuthology.orchestra.run.smithi145.stdout: "name": { 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "client", 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "num": 14775 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 753534628 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:36:02.294 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 753534628 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "inst_str": "client.14775 192.168.0.1:0/753534628", 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "addr_str": "192.168.0.1:0/753534628", 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "inode_count": 1, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_epoch": 17, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch": 91, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch_barrier": 0, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "blocklisted": false, 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs" 2024-03-28T18:36:02.295 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:36:02.300 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:02 smithi145 ceph-mon[32151]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s wr, 1 op/s 2024-03-28T18:36:02.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:02.302 DEBUG:teuthology.orchestra.run.smithi145:> stat --file-system '--printf=%T 2024-03-28T18:36:02.302 DEBUG:teuthology.orchestra.run.smithi145:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:02.307 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:02 smithi192 ceph-mon[36589]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s wr, 1 op/s 2024-03-28T18:36:02.317 INFO:teuthology.orchestra.run.smithi145.stdout:fuseblk 2024-03-28T18:36:02.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:02.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:02.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:02.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:02.386 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T18:36:02.458 INFO:teuthology.orchestra.run.smithi145.stdout:# file: . 2024-03-28T18:36:02.458 INFO:teuthology.orchestra.run.smithi145.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T18:36:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:02.459 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:36:02.459 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:36:02.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:02 smithi164 ceph-mon[36248]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s wr, 1 op/s 2024-03-28T18:36:02.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- 172.21.15.145:0/2000328349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc102170 msgr2=0x7fc9bc108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2000328349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc102170 0x7fc9bc108a00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009ca0 tx=0x7fc9b402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- 172.21.15.145:0/2000328349 shutdown_connections 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2000328349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc102170 0x7fc9bc108a00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2000328349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2000328349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc1005c0 0x7fc9bc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- 172.21.15.145:0/2000328349 >> 172.21.15.145:0/2000328349 conn(0x7fc9bc0fbb30 msgr2=0x7fc9bc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- 172.21.15.145:0/2000328349 shutdown_connections 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- 172.21.15.145:0/2000328349 wait complete. 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 Processor -- start 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.660+0000 7fc9c249a640 1 -- start start 2024-03-28T18:36:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1005c0 0x7fc9bc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc102170 0x7fc9bc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc10c240 con 0x7fc9bc1017b0 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc10c0c0 con 0x7fc9bc1005c0 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc10c3c0 con 0x7fc9bc102170 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36778/0 (socket says 172.21.15.145:36778) 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 -- 172.21.15.145:0/2953193552 learned_addr learned my addr 172.21.15.145:0/2953193552 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c0a37640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc102170 0x7fc9bc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bbfff640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1005c0 0x7fc9bc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 -- 172.21.15.145:0/2953193552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc102170 msgr2=0x7fc9bc19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc102170 0x7fc9bc19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 -- 172.21.15.145:0/2953193552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1005c0 msgr2=0x7fc9bc1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1005c0 0x7fc9bc1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 -- 172.21.15.145:0/2953193552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc19f5f0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9bb7fe640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00d950 tx=0x7fc9ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac017070 con 0x7fc9bc1017b0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9ac009c30 con 0x7fc9bc1017b0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc19f8e0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.661+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac00f6b0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.662+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc19fe30 con 0x7fc9bc1017b0 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.662+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc1a0060 con 0x7fc9bc1017b0 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.664+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc9ac0109d0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.664+0000 7fc9b97fa640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 0x7fc9a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.664+0000 7fc9bbfff640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 0x7fc9a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.664+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc9ac0ce7e0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.665+0000 7fc9bbfff640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 0x7fc9a40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc101610 tx=0x7fc9b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:02.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.668+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc9ac009de0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.851+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9bc1a02d0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.853+0000 7fc9b97fa640 1 -- 172.21.15.145:0/2953193552 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fc9ac09aff0 con 0x7fc9bc1017b0 2024-03-28T18:36:02.854 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:02.854 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 msgr2=0x7fc9a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 0x7fc9a40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc101610 tx=0x7fc9b0005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 msgr2=0x7fc9bc197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00d950 tx=0x7fc9ac00de10 comp rx=0 tx=0).stop 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 shutdown_connections 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a40a2800 0x7fc9a40a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9bc102170 0x7fc9bc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9bc1017b0 0x7fc9bc197f10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 --2- 172.21.15.145:0/2953193552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc1005c0 0x7fc9bc1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 >> 172.21.15.145:0/2953193552 conn(0x7fc9bc0fbb30 msgr2=0x7fc9bc0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:02.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 shutdown_connections 2024-03-28T18:36:02.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.855+0000 7fc9c249a640 1 -- 172.21.15.145:0/2953193552 wait complete. 2024-03-28T18:36:02.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:36:02.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:36:02.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 -- 172.21.15.145:0/2492633975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 msgr2=0x7fb7481045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/2492633975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb7481045d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009900 tx=0x7fb73c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 -- 172.21.15.145:0/2492633975 shutdown_connections 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/2492633975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748107440 0x7fb748109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/2492633975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748104b10 0x7fb748106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/2492633975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb7481045d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 -- 172.21.15.145:0/2492633975 >> 172.21.15.145:0/2492633975 conn(0x7fb7480fbb30 msgr2=0x7fb7480fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 -- 172.21.15.145:0/2492633975 shutdown_connections 2024-03-28T18:36:02.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.974+0000 7fb74c8d1640 1 -- 172.21.15.145:0/2492633975 wait complete. 2024-03-28T18:36:02.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 Processor -- start 2024-03-28T18:36:02.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 -- start start 2024-03-28T18:36:02.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748104b10 0x7fb748197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748107440 0x7fb74819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810d330 con 0x7fb7481021e0 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810d1b0 con 0x7fb748104b10 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74c8d1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810d4b0 con 0x7fb748107440 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36812/0 (socket says 172.21.15.145:36812) 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb74659c640 1 -- 172.21.15.145:0/988017651 learned_addr learned my addr 172.21.15.145:0/988017651 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.975+0000 7fb746d9d640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748107440 0x7fb74819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb745d9b640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748104b10 0x7fb748197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 -- 172.21.15.145:0/988017651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748107440 msgr2=0x7fb74819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748107440 0x7fb74819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 -- 172.21.15.145:0/988017651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748104b10 msgr2=0x7fb748197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748104b10 0x7fb748197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 -- 172.21.15.145:0/988017651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74819f630 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74659c640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009230 tx=0x7fb73c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c041070 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb73c004450 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74819f8c0 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c003e20 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.976+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74819fdb0 con 0x7fb7481021e0 2024-03-28T18:36:02.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.977+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7480fd620 con 0x7fb7481021e0 2024-03-28T18:36:02.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.978+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb73c044070 con 0x7fb7481021e0 2024-03-28T18:36:02.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.978+0000 7fb7377fe640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 0x7fb7300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:02.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.979+0000 7fb745d9b640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 0x7fb7300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:02.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.979+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb73c0f08f0 con 0x7fb7481021e0 2024-03-28T18:36:02.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.979+0000 7fb745d9b640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 0x7fb7300a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb748198f20 tx=0x7fb738005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:02.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:02.983+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb73c0bd750 con 0x7fb7481021e0 2024-03-28T18:36:03.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.183+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb74802d070 con 0x7fb7481021e0 2024-03-28T18:36:03.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.183+0000 7fb7377fe640 1 -- 172.21.15.145:0/988017651 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb73c0bd050 con 0x7fb7481021e0 2024-03-28T18:36:03.184 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:36:03.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 msgr2=0x7fb7300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 0x7fb7300a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb748198f20 tx=0x7fb738005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 msgr2=0x7fb748197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009230 tx=0x7fb73c02c650 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 shutdown_connections 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb7300a2800 0x7fb7300a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb748107440 0x7fb74819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748104b10 0x7fb748197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 --2- 172.21.15.145:0/988017651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7481021e0 0x7fb748197a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.185+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 >> 172.21.15.145:0/988017651 conn(0x7fb7480fbb30 msgr2=0x7fb748102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.186+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 shutdown_connections 2024-03-28T18:36:03.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.186+0000 7fb74c8d1640 1 -- 172.21.15.145:0/988017651 wait complete. 2024-03-28T18:36:03.196 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:03 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2953193552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:03.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 -- 172.21.15.145:0/2817644863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 msgr2=0x7fed141009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/2817644863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed141009c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fed10009900 tx=0x7fed1002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 -- 172.21.15.145:0/2817644863 shutdown_connections 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/2817644863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 0x7fed141089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/2817644863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/2817644863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed141009c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 -- 172.21.15.145:0/2817644863 >> 172.21.15.145:0/2817644863 conn(0x7fed140fbb30 msgr2=0x7fed140fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 -- 172.21.15.145:0/2817644863 shutdown_connections 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 -- 172.21.15.145:0/2817644863 wait complete. 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.302+0000 7fed1c4c6640 1 Processor -- start 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 -- start start 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 0x7fed1419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1410c230 con 0x7fed141005a0 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1410c0b0 con 0x7fed14101790 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1410c3b0 con 0x7fed14102150 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36832/0 (socket says 172.21.15.145:36832) 2024-03-28T18:36:03.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 -- 172.21.15.145:0/486698637 learned_addr learned my addr 172.21.15.145:0/486698637 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1aa63640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 0x7fed1419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed19a61640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 -- 172.21.15.145:0/486698637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 msgr2=0x7fed1419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 0x7fed1419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 -- 172.21.15.145:0/486698637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 msgr2=0x7fed14197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 -- 172.21.15.145:0/486698637 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1419f640 con 0x7fed141005a0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed19a61640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1a262640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed10005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10041070 con 0x7fed141005a0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1419f8d0 con 0x7fed141005a0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.303+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed100044d0 con 0x7fed141005a0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.304+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10034440 con 0x7fed141005a0 2024-03-28T18:36:03.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.304+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1419fe10 con 0x7fed141005a0 2024-03-28T18:36:03.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.304+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed141a0040 con 0x7fed141005a0 2024-03-28T18:36:03.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.306+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fed10044070 con 0x7fed141005a0 2024-03-28T18:36:03.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.306+0000 7fed0b7fe640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 0x7fed000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.309 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.306+0000 7fed19a61640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 0x7fed000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.309 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.306+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fed100f0a60 con 0x7fed141005a0 2024-03-28T18:36:03.309 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.307+0000 7fed19a61640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 0x7fed000a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed04005d00 tx=0x7fed04005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.310 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.310+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed100bd8c0 con 0x7fed141005a0 2024-03-28T18:36:03.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:03 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2953193552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:03.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:03 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2953193552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:03.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.501+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed141a0210 con 0x7fed141005a0 2024-03-28T18:36:03.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.501+0000 7fed0b7fe640 1 -- 172.21.15.145:0/486698637 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed100bd1c0 con 0x7fed141005a0 2024-03-28T18:36:03.502 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:36:03.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 msgr2=0x7fed000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 0x7fed000a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed04005d00 tx=0x7fed04005ed0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 msgr2=0x7fed14197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed10005d50 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 shutdown_connections 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fed000a2800 0x7fed000a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed14102150 0x7fed1419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed14101790 0x7fed14197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 --2- 172.21.15.145:0/486698637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed141005a0 0x7fed14197a70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 >> 172.21.15.145:0/486698637 conn(0x7fed140fbb30 msgr2=0x7fed140fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 shutdown_connections 2024-03-28T18:36:03.505 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.503+0000 7fed1c4c6640 1 -- 172.21.15.145:0/486698637 wait complete. 2024-03-28T18:36:03.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:36:03.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/3680772756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b881028c0 msgr2=0x7f9b88102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/3680772756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b881028c0 0x7f9b88102d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009900 tx=0x7f9b7002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/3680772756 shutdown_connections 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/3680772756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b88103280 0x7f9b88109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/3680772756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b881028c0 0x7f9b88102d40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/3680772756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 0x7f9b88101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/3680772756 >> 172.21.15.145:0/3680772756 conn(0x7f9b880fcc40 msgr2=0x7f9b880ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/3680772756 shutdown_connections 2024-03-28T18:36:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.618+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/3680772756 wait complete. 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 Processor -- start 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 -- start start 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 0x7f9b88197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 0x7f9b88197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8810d350 con 0x7f9b88103280 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8810d1d0 con 0x7f9b881028c0 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8cf3b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8810d4d0 con 0x7f9b881016d0 2024-03-28T18:36:03.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b86d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b86d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36862/0 (socket says 172.21.15.145:36862) 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b8659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 0x7f9b88197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.619+0000 7f9b86d9d640 1 -- 172.21.15.145:0/1911261718 learned_addr learned my addr 172.21.15.145:0/1911261718 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b85d9b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 0x7f9b88197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 -- 172.21.15.145:0/1911261718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 msgr2=0x7f9b88197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 0x7f9b88197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 -- 172.21.15.145:0/1911261718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 msgr2=0x7f9b88197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 0x7f9b88197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 -- 172.21.15.145:0/1911261718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8819f5b0 con 0x7f9b88103280 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b85d9b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 0x7f9b88197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b86d9d640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c970 tx=0x7f9b8000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80007bb0 con 0x7f9b88103280 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b80007d50 con 0x7f9b88103280 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80018800 con 0x7f9b88103280 2024-03-28T18:36:03.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8819f8a0 con 0x7f9b88103280 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.620+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8819fda0 con 0x7f9b88103280 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.621+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b881089f0 con 0x7f9b88103280 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.622+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9b8000fa50 con 0x7f9b88103280 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.623+0000 7f9b7f7fe640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 0x7f9b740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.623+0000 7f9b8659c640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 0x7f9b740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.624 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.623+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9b800ced30 con 0x7f9b88103280 2024-03-28T18:36:03.625 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.623+0000 7f9b8659c640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 0x7f9b740a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b88102720 tx=0x7f9b78009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.628 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.627+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b80014030 con 0x7f9b88103280 2024-03-28T18:36:03.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.800+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9b880622a0 con 0x7f9b88103280 2024-03-28T18:36:03.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.801+0000 7f9b7f7fe640 1 -- 172.21.15.145:0/1911261718 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9b8009bb90 con 0x7f9b88103280 2024-03-28T18:36:03.802 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 msgr2=0x7f9b740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 0x7f9b740a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b88102720 tx=0x7f9b78009500 comp rx=0 tx=0).stop 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 msgr2=0x7f9b8819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c970 tx=0x7f9b8000ce30 comp rx=0 tx=0).stop 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 shutdown_connections 2024-03-28T18:36:03.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b88103280 0x7f9b8819ef40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b881028c0 0x7f9b88197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b740a2800 0x7f9b740a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 --2- 172.21.15.145:0/1911261718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b881016d0 0x7f9b88197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 >> 172.21.15.145:0/1911261718 conn(0x7f9b880fcc40 msgr2=0x7f9b88105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 shutdown_connections 2024-03-28T18:36:03.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.803+0000 7f9b8cf3b640 1 -- 172.21.15.145:0/1911261718 wait complete. 2024-03-28T18:36:03.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:36:03.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 -- 172.21.15.145:0/1841726641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc104b00 msgr2=0x7fd1fc106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 --2- 172.21.15.145:0/1841726641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc104b00 0x7fd1fc106ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009900 tx=0x7fd1f002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 -- 172.21.15.145:0/1841726641 shutdown_connections 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 --2- 172.21.15.145:0/1841726641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 0x7fd1fc109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 --2- 172.21.15.145:0/1841726641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc104b00 0x7fd1fc106ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 --2- 172.21.15.145:0/1841726641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.918+0000 7fd202627640 1 -- 172.21.15.145:0/1841726641 >> 172.21.15.145:0/1841726641 conn(0x7fd1fc0fbb40 msgr2=0x7fd1fc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- 172.21.15.145:0/1841726641 shutdown_connections 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- 172.21.15.145:0/1841726641 wait complete. 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 Processor -- start 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- start start 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc104b00 0x7fd1fc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 0x7fd1fc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc10d320 con 0x7fd1fc1021d0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc10d1a0 con 0x7fd1fc107430 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd202627640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc10d4a0 con 0x7fd1fc104b00 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.919+0000 7fd1fbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36890/0 (socket says 172.21.15.145:36890) 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 -- 172.21.15.145:0/980907504 learned_addr learned my addr 172.21.15.145:0/980907504 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fb7fe640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc104b00 0x7fd1fc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd200bc4640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 0x7fd1fc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 -- 172.21.15.145:0/980907504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc104b00 msgr2=0x7fd1fc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc104b00 0x7fd1fc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 -- 172.21.15.145:0/980907504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 msgr2=0x7fd1fc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 0x7fd1fc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 -- 172.21.15.145:0/980907504 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc19f650 con 0x7fd1fc1021d0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1fbfff640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e9f0 tx=0x7fd1ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec00cd80 con 0x7fd1fc1021d0 2024-03-28T18:36:03.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1ec005500 con 0x7fd1fc1021d0 2024-03-28T18:36:03.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1fc19f940 con 0x7fd1fc1021d0 2024-03-28T18:36:03.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec010690 con 0x7fd1fc1021d0 2024-03-28T18:36:03.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.920+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1fc19fe10 con 0x7fd1fc1021d0 2024-03-28T18:36:03.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.921+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1fc0fd630 con 0x7fd1fc1021d0 2024-03-28T18:36:03.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.922+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd1ec0046c0 con 0x7fd1fc1021d0 2024-03-28T18:36:03.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.923+0000 7fd1f97fa640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:03.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.923+0000 7fd1fb7fe640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:03.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.923+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd1ec0cf0e0 con 0x7fd1fc1021d0 2024-03-28T18:36:03.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.923+0000 7fd1fb7fe640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 0x7fd1e40a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0000c00 tx=0x7fd1f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:03.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:03.927+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd1ec09bee0 con 0x7fd1fc1021d0 2024-03-28T18:36:04.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.103+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd1fc0622d0 con 0x7fd1fc1021d0 2024-03-28T18:36:04.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.103+0000 7fd1f97fa640 1 -- 172.21.15.145:0/980907504 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd1ec09b7e0 con 0x7fd1fc1021d0 2024-03-28T18:36:04.104 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 msgr2=0x7fd1e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 0x7fd1e40a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0000c00 tx=0x7fd1f0005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 msgr2=0x7fd1fc197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00e9f0 tx=0x7fd1ec00edf0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 shutdown_connections 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1fc107430 0x7fd1fc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1fc104b00 0x7fd1fc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 --2- 172.21.15.145:0/980907504 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1fc1021d0 0x7fd1fc197a30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 >> 172.21.15.145:0/980907504 conn(0x7fd1fc0fbb40 msgr2=0x7fd1fc102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 shutdown_connections 2024-03-28T18:36:04.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.106+0000 7fd202627640 1 -- 172.21.15.145:0/980907504 wait complete. 2024-03-28T18:36:04.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/988017651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.116 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:04 smithi145 ceph-mon[32151]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 362 B/s wr, 0 op/s 2024-03-28T18:36:04.117 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/486698637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.117 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1911261718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- 172.21.15.145:0/1830970411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 msgr2=0x7f77f01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/1830970411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f01009e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009900 tx=0x7f77e802bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- 172.21.15.145:0/1830970411 shutdown_connections 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/1830970411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/1830970411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 0x7f77f0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/1830970411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f01009e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- 172.21.15.145:0/1830970411 >> 172.21.15.145:0/1830970411 conn(0x7f77f00fbb50 msgr2=0x7f77f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- 172.21.15.145:0/1830970411 shutdown_connections 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- 172.21.15.145:0/1830970411 wait complete. 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 Processor -- start 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.220+0000 7f77f5a3d640 1 -- start start 2024-03-28T18:36:04.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 0x7f77f0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f010c210 con 0x7f77f01005c0 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f010c090 con 0x7f77f0102170 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f010c390 con 0x7f77f01017b0 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36908/0 (socket says 172.21.15.145:36908) 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 -- 172.21.15.145:0/665088920 learned_addr learned my addr 172.21.15.145:0/665088920 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77ee7fc640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 0x7f77f0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77ef7fe640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 -- 172.21.15.145:0/665088920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 msgr2=0x7f77f0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 0x7f77f0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 -- 172.21.15.145:0/665088920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 msgr2=0x7f77f019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 -- 172.21.15.145:0/665088920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f019f680 con 0x7f77f01005c0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77ef7fe640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77eeffd640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77e8000c00 tx=0x7f77e8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8041070 con 0x7f77f01005c0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77e80044d0 con 0x7f77f01005c0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f019f910 con 0x7f77f01005c0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8034440 con 0x7f77f01005c0 2024-03-28T18:36:04.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.221+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f019fe00 con 0x7f77f01005c0 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.222+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f01a0030 con 0x7f77f01005c0 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.223+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f77e8044070 con 0x7f77f01005c0 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.224+0000 7f77f4a3b640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.224+0000 7f77ee7fc640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.224+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f77e80f0a60 con 0x7f77f01005c0 2024-03-28T18:36:04.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.224+0000 7f77ee7fc640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77d80095f0 tx=0x7f77d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.227+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f77e80bd8c0 con 0x7f77f01005c0 2024-03-28T18:36:04.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/988017651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:04 smithi192 ceph-mon[36589]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 362 B/s wr, 0 op/s 2024-03-28T18:36:04.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/486698637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1911261718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.417 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.416+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f77f01a0200 con 0x7f77f01005c0 2024-03-28T18:36:04.417 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.416+0000 7f77f4a3b640 1 -- 172.21.15.145:0/665088920 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+600 (secure 0 0 0) 0x7f77e80bd1c0 con 0x7f77f01005c0 2024-03-28T18:36:04.417 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/606368710 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:04.417 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/4006486926 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/2249018729 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3480949122 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/832150839 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3638297666 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:04.418 INFO:teuthology.orchestra.run.smithi145.stderr:listed 10 entries 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.418+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 msgr2=0x7f77dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.418+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f77d80095f0 tx=0x7f77d8005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.418+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 msgr2=0x7f77f0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.418+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77e8000c00 tx=0x7f77e8005d50 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 shutdown_connections 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77f0102170 0x7f77f019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77dc0a2800 0x7f77dc0a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77f01017b0 0x7f77f0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 --2- 172.21.15.145:0/665088920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f01005c0 0x7f77f0197a60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 >> 172.21.15.145:0/665088920 conn(0x7f77f00fbb50 msgr2=0x7f77f0069d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 shutdown_connections 2024-03-28T18:36:04.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.419+0000 7f77f5a3d640 1 -- 172.21.15.145:0/665088920 wait complete. 2024-03-28T18:36:04.430 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:36:04.430 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:36:04.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:36:04.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/988017651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:04 smithi164 ceph-mon[36248]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 362 B/s wr, 0 op/s 2024-03-28T18:36:04.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/486698637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1911261718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 -- 172.21.15.145:0/2468085664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 msgr2=0x7f15a4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 --2- 172.21.15.145:0/2468085664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4106f20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1594009900 tx=0x7f159402bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 -- 172.21.15.145:0/2468085664 shutdown_connections 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 --2- 172.21.15.145:0/2468085664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4107460 0x7f15a4109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 --2- 172.21.15.145:0/2468085664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4106f20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.590 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 --2- 172.21.15.145:0/2468085664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4102200 0x7f15a41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 -- 172.21.15.145:0/2468085664 >> 172.21.15.145:0/2468085664 conn(0x7f15a40fbb50 msgr2=0x7f15a40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 -- 172.21.15.145:0/2468085664 shutdown_connections 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.587+0000 7f15ac027640 1 -- 172.21.15.145:0/2468085664 wait complete. 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 Processor -- start 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 -- start start 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 0x7f15a4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4107460 0x7f15a419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a410d330 con 0x7f15a4104b30 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a410d1b0 con 0x7f15a4107460 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15ac027640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a410d4b0 con 0x7f15a4102200 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36930/0 (socket says 172.21.15.145:36930) 2024-03-28T18:36:04.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 -- 172.21.15.145:0/134057249 learned_addr learned my addr 172.21.15.145:0/134057249 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a9dc3640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 0x7f15a4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15aa5c4640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4107460 0x7f15a419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 -- 172.21.15.145:0/134057249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 msgr2=0x7f15a4197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 0x7f15a4197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 -- 172.21.15.145:0/134057249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4107460 msgr2=0x7f15a419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4107460 0x7f15a419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 -- 172.21.15.145:0/134057249 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a419f620 con 0x7f15a4104b30 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a9dc3640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 0x7f15a4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f15a95c2640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1594000c00 tx=0x7f1594005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.588+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594041070 con 0x7f15a4104b30 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.589+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a419f8b0 con 0x7f15a4104b30 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.589+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15940044d0 con 0x7f15a4104b30 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.589+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594034440 con 0x7f15a4104b30 2024-03-28T18:36:04.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.589+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a419fda0 con 0x7f15a4104b30 2024-03-28T18:36:04.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.590+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a40fd640 con 0x7f15a4104b30 2024-03-28T18:36:04.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.591+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1594044070 con 0x7f15a4104b30 2024-03-28T18:36:04.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.591+0000 7f159affd640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 0x7f15900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.594 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.591+0000 7f15a9dc3640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 0x7f15900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.594 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.591+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f15940f08f0 con 0x7f15a4104b30 2024-03-28T18:36:04.594 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.592+0000 7f15a9dc3640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 0x7f15900a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f15a0005d00 tx=0x7f15a0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.596 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.596+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15940bd750 con 0x7f15a4104b30 2024-03-28T18:36:04.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.772+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15a40622a0 con 0x7f15a4104b30 2024-03-28T18:36:04.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.774+0000 7f159affd640 1 -- 172.21.15.145:0/134057249 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7f15940bd050 con 0x7f15a4104b30 2024-03-28T18:36:04.775 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:04.775 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 msgr2=0x7f15900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 0x7f15900a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f15a0005d00 tx=0x7f15a0005c70 comp rx=0 tx=0).stop 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 msgr2=0x7f15a4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1594000c00 tx=0x7f1594005d50 comp rx=0 tx=0).stop 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 shutdown_connections 2024-03-28T18:36:04.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a4107460 0x7f15a419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15a4104b30 0x7f15a4197f40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f15900a2800 0x7f15900a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 --2- 172.21.15.145:0/134057249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15a4102200 0x7f15a4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 >> 172.21.15.145:0/134057249 conn(0x7f15a40fbb50 msgr2=0x7f15a4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 shutdown_connections 2024-03-28T18:36:04.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.776+0000 7f15ac027640 1 -- 172.21.15.145:0/134057249 wait complete. 2024-03-28T18:36:04.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:36:04.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 -- 172.21.15.145:0/3912093197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20104ad0 msgr2=0x7f2f20106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 --2- 172.21.15.145:0/3912093197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20104ad0 0x7f2f20106ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0802bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 -- 172.21.15.145:0/3912093197 shutdown_connections 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 --2- 172.21.15.145:0/3912093197 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20107400 0x7f2f201097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 --2- 172.21.15.145:0/3912093197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20104ad0 0x7f2f20106ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 --2- 172.21.15.145:0/3912093197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f201021a0 0x7f2f20104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 -- 172.21.15.145:0/3912093197 >> 172.21.15.145:0/3912093197 conn(0x7f2f200fbb30 msgr2=0x7f2f200fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 -- 172.21.15.145:0/3912093197 shutdown_connections 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.892+0000 7f2f25932640 1 -- 172.21.15.145:0/3912093197 wait complete. 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 Processor -- start 2024-03-28T18:36:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 -- start start 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20104ad0 0x7f2f20197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f20107400 0x7f2f2019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2010d300 con 0x7f2f201021a0 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2010d180 con 0x7f2f20107400 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f25932640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2010d480 con 0x7f2f20104ad0 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f1effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36942/0 (socket says 172.21.15.145:36942) 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.893+0000 7f2f1effd640 1 -- 172.21.15.145:0/519911732 learned_addr learned my addr 172.21.15.145:0/519911732 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1e7fc640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20104ad0 0x7f2f20197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 -- 172.21.15.145:0/519911732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20104ad0 msgr2=0x7f2f20197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20104ad0 0x7f2f20197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1f7fe640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f20107400 0x7f2f2019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 -- 172.21.15.145:0/519911732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f20107400 msgr2=0x7f2f2019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f20107400 0x7f2f2019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 -- 172.21.15.145:0/519911732 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2019f650 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f1effd640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000ebc0 tx=0x7f2f1000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10010630 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2019f940 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f10010cb0 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10018bd0 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.894+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2019feb0 con 0x7f2f201021a0 2024-03-28T18:36:04.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.895+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f200fd620 con 0x7f2f201021a0 2024-03-28T18:36:04.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.896+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2f100107d0 con 0x7f2f201021a0 2024-03-28T18:36:04.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.897+0000 7f2f24930640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 0x7f2f0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:04.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.897+0000 7f2f1e7fc640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 0x7f2f0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:04.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.897+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2f100cf330 con 0x7f2f201021a0 2024-03-28T18:36:04.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.897+0000 7f2f1e7fc640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 0x7f2f0c0a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:04.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:04.901+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f1009c190 con 0x7f2f201021a0 2024-03-28T18:36:05.075 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.074+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f200622d0 con 0x7f2f201021a0 2024-03-28T18:36:05.075 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.075+0000 7f2f24930640 1 -- 172.21.15.145:0/519911732 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f1009ba90 con 0x7f2f201021a0 2024-03-28T18:36:05.076 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 msgr2=0x7f2f0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 0x7f2f0c0a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2f08000c00 tx=0x7f2f08005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 msgr2=0x7f2f20197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000ebc0 tx=0x7f2f1000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 shutdown_connections 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f20107400 0x7f2f2019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2f0c0a2800 0x7f2f0c0a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f20104ad0 0x7f2f20197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 --2- 172.21.15.145:0/519911732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f201021a0 0x7f2f20197a30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 >> 172.21.15.145:0/519911732 conn(0x7f2f200fbb30 msgr2=0x7f2f20102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 shutdown_connections 2024-03-28T18:36:05.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.077+0000 7f2f25932640 1 -- 172.21.15.145:0/519911732 wait complete. 2024-03-28T18:36:05.088 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- 172.21.15.145:0/481863988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58201016b0 msgr2=0x7f5820101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 --2- 172.21.15.145:0/481863988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58201016b0 0x7f5820101ad0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5814009900 tx=0x7f581402bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- 172.21.15.145:0/481863988 shutdown_connections 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 --2- 172.21.15.145:0/481863988 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5820103260 0x7f5820109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 --2- 172.21.15.145:0/481863988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 --2- 172.21.15.145:0/481863988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58201016b0 0x7f5820101ad0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- 172.21.15.145:0/481863988 >> 172.21.15.145:0/481863988 conn(0x7f58200fcc40 msgr2=0x7f58200ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- 172.21.15.145:0/481863988 shutdown_connections 2024-03-28T18:36:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- 172.21.15.145:0/481863988 wait complete. 2024-03-28T18:36:05.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 Processor -- start 2024-03-28T18:36:05.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.192+0000 7f5824b5b640 1 -- start start 2024-03-28T18:36:05.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58201016b0 0x7f5820075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582010d300 con 0x7f5820103260 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582010d180 con 0x7f58201028a0 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582010d480 con 0x7f58201016b0 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36958/0 (socket says 172.21.15.145:36958) 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 -- 172.21.15.145:0/1374620734 learned_addr learned my addr 172.21.15.145:0/1374620734 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581e59c640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58201016b0 0x7f5820075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581dd9b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 -- 172.21.15.145:0/1374620734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58201016b0 msgr2=0x7f5820075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58201016b0 0x7f5820075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 -- 172.21.15.145:0/1374620734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 msgr2=0x7f5820073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 -- 172.21.15.145:0/1374620734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820074860 con 0x7f5820103260 2024-03-28T18:36:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581dd9b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f581ed9d640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f581800eb10 tx=0x7f581800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818004b20 con 0x7f5820103260 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5818004cc0 con 0x7f5820103260 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58201a1b20 con 0x7f5820103260 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818010760 con 0x7f5820103260 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.193+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58201a2020 con 0x7f5820103260 2024-03-28T18:36:05.197 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.194+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5820108a40 con 0x7f5820103260 2024-03-28T18:36:05.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.195+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5818005020 con 0x7f5820103260 2024-03-28T18:36:05.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.196+0000 7f580f7fe640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 0x7f58080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.196+0000 7f581e59c640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 0x7f58080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.196+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58180cf330 con 0x7f5820103260 2024-03-28T18:36:05.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.196+0000 7f581e59c640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 0x7f58080a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5814000c00 tx=0x7f5814005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.201 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.200+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f581809c190 con 0x7f5820103260 2024-03-28T18:36:05.375 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.374+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5820062340 con 0x7f5820103260 2024-03-28T18:36:05.376 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.375+0000 7f580f7fe640 1 -- 172.21.15.145:0/1374620734 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f581809ba90 con 0x7f5820103260 2024-03-28T18:36:05.376 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 msgr2=0x7f58080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 0x7f58080a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5814000c00 tx=0x7f5814005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 msgr2=0x7f5820073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f581800eb10 tx=0x7f581800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 shutdown_connections 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5820103260 0x7f5820073fd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58201028a0 0x7f5820073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f58080a2800 0x7f58080a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 --2- 172.21.15.145:0/1374620734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58201016b0 0x7f5820075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 >> 172.21.15.145:0/1374620734 conn(0x7f58200fcc40 msgr2=0x7f5820105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 shutdown_connections 2024-03-28T18:36:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.377+0000 7f5824b5b640 1 -- 172.21.15.145:0/1374620734 wait complete. 2024-03-28T18:36:05.387 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/980907504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:05.388 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/665088920' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:05.388 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/134057249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:36:05.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/980907504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:05.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/665088920' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:05.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/134057249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/980907504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/665088920' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/134057249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 -- 172.21.15.145:0/1544791541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f8103260 msgr2=0x7f13f8109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 --2- 172.21.15.145:0/1544791541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f8103260 0x7f13f8109af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009ca0 tx=0x7f13f002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 -- 172.21.15.145:0/1544791541 shutdown_connections 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 --2- 172.21.15.145:0/1544791541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f8103260 0x7f13f8109af0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 --2- 172.21.15.145:0/1544791541 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f81028a0 0x7f13f8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 --2- 172.21.15.145:0/1544791541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 -- 172.21.15.145:0/1544791541 >> 172.21.15.145:0/1544791541 conn(0x7f13f80fcc40 msgr2=0x7f13f80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 -- 172.21.15.145:0/1544791541 shutdown_connections 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.493+0000 7f13fdc59640 1 -- 172.21.15.145:0/1544791541 wait complete. 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 Processor -- start 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 -- start start 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f8103260 0x7f13f819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f810d310 con 0x7f13f81028a0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f810d190 con 0x7f13f8103260 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13fdc59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f810d490 con 0x7f13f81016b0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36966/0 (socket says 172.21.15.145:36966) 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 -- 172.21.15.145:0/3262253109 learned_addr learned my addr 172.21.15.145:0/3262253109 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13f77fe640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13f7fff640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f8103260 0x7f13f819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 -- 172.21.15.145:0/3262253109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 msgr2=0x7f13f8197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 -- 172.21.15.145:0/3262253109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f8103260 msgr2=0x7f13f819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f8103260 0x7f13f819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13effff640 1 -- 172.21.15.145:0/3262253109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f819f570 con 0x7f13f81028a0 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.494+0000 7f13f77fe640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:05.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13effff640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f13e000d950 tx=0x7f13e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e0017030 con 0x7f13f81028a0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f819f860 con 0x7f13f81028a0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13e0004020 con 0x7f13f81028a0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e00047a0 con 0x7f13f81028a0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.495+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f819fe00 con 0x7f13f81028a0 2024-03-28T18:36:05.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.496+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f81089d0 con 0x7f13f81028a0 2024-03-28T18:36:05.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.497+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f13e00041d0 con 0x7f13f81028a0 2024-03-28T18:36:05.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.497+0000 7f13f57fa640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 0x7f13e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.497+0000 7f13f77fe640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 0x7f13e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.497+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f13e00ce680 con 0x7f13f81028a0 2024-03-28T18:36:05.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.498+0000 7f13f77fe640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 0x7f13e40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f13f8102700 tx=0x7f13e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.502 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.502+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13e009b4e0 con 0x7f13f81028a0 2024-03-28T18:36:05.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.678+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13f80622a0 con 0x7f13f81028a0 2024-03-28T18:36:05.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.679+0000 7f13f57fa640 1 -- 172.21.15.145:0/3262253109 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13e009ade0 con 0x7f13f81028a0 2024-03-28T18:36:05.680 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 msgr2=0x7f13e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 0x7f13e40a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f13f8102700 tx=0x7f13e8005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 msgr2=0x7f13f8197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f13e000d950 tx=0x7f13e000de10 comp rx=0 tx=0).stop 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 shutdown_connections 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f8103260 0x7f13f819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f81028a0 0x7f13f8197e90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e40a2800 0x7f13e40a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 --2- 172.21.15.145:0/3262253109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f81016b0 0x7f13f8197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 >> 172.21.15.145:0/3262253109 conn(0x7f13f80fcc40 msgr2=0x7f13f8105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 shutdown_connections 2024-03-28T18:36:05.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.681+0000 7f13fdc59640 1 -- 172.21.15.145:0/3262253109 wait complete. 2024-03-28T18:36:05.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:36:05.797 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 -- 172.21.15.145:0/3107889606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd804103260 msgr2=0x7fd804109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 --2- 172.21.15.145:0/3107889606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd804103260 0x7fd804109af0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009ca0 tx=0x7fd7fc02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 -- 172.21.15.145:0/3107889606 shutdown_connections 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 --2- 172.21.15.145:0/3107889606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd804103260 0x7fd804109af0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 --2- 172.21.15.145:0/3107889606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8041028a0 0x7fd804102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 --2- 172.21.15.145:0/3107889606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 0x7fd804101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 -- 172.21.15.145:0/3107889606 >> 172.21.15.145:0/3107889606 conn(0x7fd8040fcc40 msgr2=0x7fd8040ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 -- 172.21.15.145:0/3107889606 shutdown_connections 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.796+0000 7fd80a377640 1 -- 172.21.15.145:0/3107889606 wait complete. 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 Processor -- start 2024-03-28T18:36:05.798 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 -- start start 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 0x7fd804075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 0x7fd804073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80410d310 con 0x7fd8041028a0 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80410d190 con 0x7fd804103260 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd80a377640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80410d490 con 0x7fd8041016b0 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd8037fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd8037fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36978/0 (socket says 172.21.15.145:36978) 2024-03-28T18:36:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd8037fe640 1 -- 172.21.15.145:0/141312124 learned_addr learned my addr 172.21.15.145:0/141312124 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd803fff640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 0x7fd804075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd808914640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 0x7fd804073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.797+0000 7fd8037fe640 1 -- 172.21.15.145:0/141312124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 msgr2=0x7fd804075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8037fe640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 0x7fd804075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8037fe640 1 -- 172.21.15.145:0/141312124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 msgr2=0x7fd804073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8037fe640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 0x7fd804073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8037fe640 1 -- 172.21.15.145:0/141312124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804074860 con 0x7fd8041028a0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd808914640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 0x7fd804073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8037fe640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800d950 tx=0x7fd7f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8017030 con 0x7fd8041028a0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7f8004020 con 0x7fd8041028a0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8041a1a80 con 0x7fd8041028a0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80047a0 con 0x7fd8041028a0 2024-03-28T18:36:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.798+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8041a1fd0 con 0x7fd8041028a0 2024-03-28T18:36:05.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.799+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8041089d0 con 0x7fd8041028a0 2024-03-28T18:36:05.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.800+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd7f80041d0 con 0x7fd8041028a0 2024-03-28T18:36:05.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.801+0000 7fd8017fa640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 0x7fd7ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:05.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.801+0000 7fd803fff640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 0x7fd7ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:05.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.801+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd7f80ce6f0 con 0x7fd8041028a0 2024-03-28T18:36:05.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.801+0000 7fd803fff640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 0x7fd7ec0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40095f0 tx=0x7fd7f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:05.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.805+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd7f809b550 con 0x7fd8041028a0 2024-03-28T18:36:05.982 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.981+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd8040622a0 con 0x7fd8041028a0 2024-03-28T18:36:05.982 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.981+0000 7fd8017fa640 1 -- 172.21.15.145:0/141312124 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd7f809ae50 con 0x7fd8041028a0 2024-03-28T18:36:05.982 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:36:05.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 msgr2=0x7fd7ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 0x7fd7ec0a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40095f0 tx=0x7fd7f4005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:05.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 msgr2=0x7fd804073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800d950 tx=0x7fd7f800de10 comp rx=0 tx=0).stop 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 shutdown_connections 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd804103260 0x7fd804073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8041028a0 0x7fd804073a90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd7ec0a2800 0x7fd7ec0a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 --2- 172.21.15.145:0/141312124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8041016b0 0x7fd804075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 >> 172.21.15.145:0/141312124 conn(0x7fd8040fcc40 msgr2=0x7fd804105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 shutdown_connections 2024-03-28T18:36:05.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:05.983+0000 7fd80a377640 1 -- 172.21.15.145:0/141312124 wait complete. 2024-03-28T18:36:05.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T18:36:06.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 -- 172.21.15.145:0/2745296526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c1016b0 msgr2=0x7f7a2c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 --2- 172.21.15.145:0/2745296526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c101ad0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009900 tx=0x7f7a2802bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:06.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 -- 172.21.15.145:0/2745296526 shutdown_connections 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 --2- 172.21.15.145:0/2745296526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c103260 0x7f7a2c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 --2- 172.21.15.145:0/2745296526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 --2- 172.21.15.145:0/2745296526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c101ad0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 -- 172.21.15.145:0/2745296526 >> 172.21.15.145:0/2745296526 conn(0x7f7a2c0fcc40 msgr2=0x7f7a2c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 -- 172.21.15.145:0/2745296526 shutdown_connections 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.098+0000 7f7a338ff640 1 -- 172.21.15.145:0/2745296526 wait complete. 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 Processor -- start 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 -- start start 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10d3a0 con 0x7f7a2c103260 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10d220 con 0x7f7a2c1028a0 2024-03-28T18:36:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a338ff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c10d520 con 0x7f7a2c1016b0 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37004/0 (socket says 172.21.15.145:37004) 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 -- 172.21.15.145:0/1939556641 learned_addr learned my addr 172.21.15.145:0/1939556641 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a3169b640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a30e9a640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 -- 172.21.15.145:0/1939556641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1016b0 msgr2=0x7f7a2c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 -- 172.21.15.145:0/1939556641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 msgr2=0x7f7a2c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.099+0000 7f7a31e9c640 1 -- 172.21.15.145:0/1939556641 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c1a1920 con 0x7f7a2c103260 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a30e9a640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a31e9c640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400b970 tx=0x7f7a2400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24019070 con 0x7f7a2c103260 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a2400cb40 con 0x7f7a2c103260 2024-03-28T18:36:06.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c1a1c10 con 0x7f7a2c103260 2024-03-28T18:36:06.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24007650 con 0x7f7a2c103260 2024-03-28T18:36:06.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.100+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1a2130 con 0x7f7a2c103260 2024-03-28T18:36:06.104 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.101+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c1a2360 con 0x7f7a2c103260 2024-03-28T18:36:06.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.102+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7a2401c070 con 0x7f7a2c103260 2024-03-28T18:36:06.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.103+0000 7f7a227fc640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 0x7f7a180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.103+0000 7f7a3169b640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 0x7f7a180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.103+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7a240ce1e0 con 0x7f7a2c103260 2024-03-28T18:36:06.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.103+0000 7f7a3169b640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 0x7f7a180a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7a28000c00 tx=0x7f7a28005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.107+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a2409aa90 con 0x7f7a2c103260 2024-03-28T18:36:06.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:06 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/519911732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:06 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1374620734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:06 smithi145 ceph-mon[32151]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T18:36:06.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:06 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3262253109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:06 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/141312124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.296+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7a2c1a2530 con 0x7f7a2c103260 2024-03-28T18:36:06.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.296+0000 7f7a227fc640 1 -- 172.21.15.145:0/1939556641 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+600 (secure 0 0 0) 0x7f7a2409eb30 con 0x7f7a2c103260 2024-03-28T18:36:06.297 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/606368710 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/4006486926 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/2249018729 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3480949122 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/832150839 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3638297666 2024-03-29T18:28:53.394017+0000 2024-03-28T18:36:06.298 INFO:teuthology.orchestra.run.smithi145.stderr:listed 10 entries 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 msgr2=0x7f7a180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 0x7f7a180a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7a28000c00 tx=0x7f7a28005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 msgr2=0x7f7a2c1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400b970 tx=0x7f7a2400be30 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 shutdown_connections 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a2c103260 0x7f7a2c1a1280 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c1028a0 0x7f7a2c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7a180a2880 0x7f7a180a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 --2- 172.21.15.145:0/1939556641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a2c1016b0 0x7f7a2c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 >> 172.21.15.145:0/1939556641 conn(0x7f7a2c0fcc40 msgr2=0x7f7a2c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 shutdown_connections 2024-03-28T18:36:06.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.299+0000 7f7a338ff640 1 -- 172.21.15.145:0/1939556641 wait complete. 2024-03-28T18:36:06.310 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi145.front.sepia.ceph.com... 2024-03-28T18:36:06.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:06.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T18:36:06.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi145.stderr:ceph-fuse[73276]: fuse finished with error 0 and tester_r 0 2024-03-28T18:36:06.338 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:36:06.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:36:06.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:36:06.394 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T18:36:06.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:06 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/519911732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:06 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1374620734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:06 smithi192 ceph-mon[36589]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T18:36:06.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:06 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3262253109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:06 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/141312124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:36:06.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:06 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/519911732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:06 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1374620734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:06 smithi164 ceph-mon[36248]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T18:36:06.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:06 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3262253109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:06 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/141312124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:36:06.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 -- 172.21.15.145:0/1938007634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8102200 msgr2=0x7f90d81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 --2- 172.21.15.145:0/1938007634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8102200 0x7f90d81045f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009900 tx=0x7f90cc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:06.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 -- 172.21.15.145:0/1938007634 shutdown_connections 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 --2- 172.21.15.145:0/1938007634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8107460 0x7f90d8109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 --2- 172.21.15.145:0/1938007634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90d8104b30 0x7f90d8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 --2- 172.21.15.145:0/1938007634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8102200 0x7f90d81045f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 -- 172.21.15.145:0/1938007634 >> 172.21.15.145:0/1938007634 conn(0x7f90d80fbb50 msgr2=0x7f90d80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 -- 172.21.15.145:0/1938007634 shutdown_connections 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.555+0000 7f90df0f1640 1 -- 172.21.15.145:0/1938007634 wait complete. 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 Processor -- start 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 -- start start 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8102200 0x7f90d8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90d8107460 0x7f90d819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d810d350 con 0x7f90d8104b30 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d810d1d0 con 0x7f90d8107460 2024-03-28T18:36:06.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90df0f1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d810d4d0 con 0x7f90d8102200 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90d7fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37016/0 (socket says 172.21.15.145:37016) 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90d7fff640 1 -- 172.21.15.145:0/844703002 learned_addr learned my addr 172.21.15.145:0/844703002 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90dce8d640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8102200 0x7f90d8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90d7fff640 1 -- 172.21.15.145:0/844703002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8102200 msgr2=0x7f90d8197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.556+0000 7f90d7fff640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8102200 0x7f90d8197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d7fff640 1 -- 172.21.15.145:0/844703002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90d8107460 msgr2=0x7f90d819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d7fff640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90d8107460 0x7f90d819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d7fff640 1 -- 172.21.15.145:0/844703002 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d819f640 con 0x7f90d8104b30 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d7fff640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90c4005fd0 tx=0x7f90c400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4017070 con 0x7f90d8104b30 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90c4009c30 con 0x7f90d8104b30 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d819f930 con 0x7f90d8104b30 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4010460 con 0x7f90d8104b30 2024-03-28T18:36:06.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.557+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d819fe80 con 0x7f90d8104b30 2024-03-28T18:36:06.561 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.558+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d80fd640 con 0x7f90d8104b30 2024-03-28T18:36:06.561 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.559+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f90c4003e70 con 0x7f90d8104b30 2024-03-28T18:36:06.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.559+0000 7f90d5ffb640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 0x7f90c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.560+0000 7f90dce8d640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 0x7f90c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.560+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f90c40cea00 con 0x7f90d8104b30 2024-03-28T18:36:06.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.560+0000 7f90dce8d640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 0x7f90c80a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90cc02c7b0 tx=0x7f90cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.564+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90c4020030 con 0x7f90d8104b30 2024-03-28T18:36:06.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.745+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90d80622a0 con 0x7f90d8104b30 2024-03-28T18:36:06.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.746+0000 7f90d5ffb640 1 -- 172.21.15.145:0/844703002 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7f90c409b860 con 0x7f90d8104b30 2024-03-28T18:36:06.747 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:06.747 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:06.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 msgr2=0x7f90c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 0x7f90c80a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90cc02c7b0 tx=0x7f90cc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 msgr2=0x7f90d8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90c4005fd0 tx=0x7f90c400dde0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 shutdown_connections 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90d8107460 0x7f90d819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90d8104b30 0x7f90d8197f60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f90c80a2800 0x7f90c80a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 --2- 172.21.15.145:0/844703002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90d8102200 0x7f90d8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 >> 172.21.15.145:0/844703002 conn(0x7f90d80fbb50 msgr2=0x7f90d8102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 shutdown_connections 2024-03-28T18:36:06.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.748+0000 7f90df0f1640 1 -- 172.21.15.145:0/844703002 wait complete. 2024-03-28T18:36:06.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- 172.21.15.145:0/2763953282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5681005c0 msgr2=0x7fd5681009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 --2- 172.21.15.145:0/2763953282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5681005c0 0x7fd5681009e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd550009900 tx=0x7fd55002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- 172.21.15.145:0/2763953282 shutdown_connections 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 --2- 172.21.15.145:0/2763953282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd568102170 0x7fd568108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 --2- 172.21.15.145:0/2763953282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 0x7fd568101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 --2- 172.21.15.145:0/2763953282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5681005c0 0x7fd5681009e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- 172.21.15.145:0/2763953282 >> 172.21.15.145:0/2763953282 conn(0x7fd5680fbb30 msgr2=0x7fd5680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- 172.21.15.145:0/2763953282 shutdown_connections 2024-03-28T18:36:06.863 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- 172.21.15.145:0/2763953282 wait complete. 2024-03-28T18:36:06.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 Processor -- start 2024-03-28T18:36:06.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.862+0000 7fd56e69a640 1 -- start start 2024-03-28T18:36:06.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5681005c0 0x7fd5681968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 0x7fd568196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56810c210 con 0x7fd568102170 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56810c090 con 0x7fd5681017b0 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56810c390 con 0x7fd5681005c0 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37034/0 (socket says 172.21.15.145:37034) 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 -- 172.21.15.145:0/237008476 learned_addr learned my addr 172.21.15.145:0/237008476 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd567fff640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5681005c0 0x7fd5681968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd5677fe640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 0x7fd568196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 -- 172.21.15.145:0/237008476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5681005c0 msgr2=0x7fd5681968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5681005c0 0x7fd5681968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 -- 172.21.15.145:0/237008476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 msgr2=0x7fd568196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:06.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 0x7fd568196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 -- 172.21.15.145:0/237008476 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56819e500 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56cc37640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd56000eca0 tx=0x7fd56000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560004b20 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd560004cc0 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd568068860 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560017650 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.863+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd568068db0 con 0x7fd568102170 2024-03-28T18:36:06.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.864+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd568068490 con 0x7fd568102170 2024-03-28T18:36:06.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.865+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd560005000 con 0x7fd568102170 2024-03-28T18:36:06.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.866+0000 7fd5657fa640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:06.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.866+0000 7fd567fff640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:06.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.866+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd5600d0e60 con 0x7fd568102170 2024-03-28T18:36:06.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.866+0000 7fd567fff640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 0x7fd5540a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd550000c00 tx=0x7fd550005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:06.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:06.870+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd56009d670 con 0x7fd568102170 2024-03-28T18:36:07.047 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.046+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd56819e790 con 0x7fd568102170 2024-03-28T18:36:07.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.047+0000 7fd5657fa640 1 -- 172.21.15.145:0/237008476 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fd5600a1710 con 0x7fd568102170 2024-03-28T18:36:07.049 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:07.049 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.049+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 msgr2=0x7fd5540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 0x7fd5540a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd550000c00 tx=0x7fd550005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 msgr2=0x7fd56819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd56000eca0 tx=0x7fd56000b690 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 shutdown_connections 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd568102170 0x7fd56819de90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5681017b0 0x7fd568196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 --2- 172.21.15.145:0/237008476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5681005c0 0x7fd5681968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 >> 172.21.15.145:0/237008476 conn(0x7fd5680fbb30 msgr2=0x7fd568104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 shutdown_connections 2024-03-28T18:36:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.050+0000 7fd56e69a640 1 -- 172.21.15.145:0/237008476 wait complete. 2024-03-28T18:36:07.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- 172.21.15.145:0/3546348474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c1033d0 msgr2=0x7f821c109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- 172.21.15.145:0/3546348474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c1033d0 0x7f821c109c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8214009ca0 tx=0x7f821402c140 comp rx=0 tx=0).stop 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- 172.21.15.145:0/3546348474 shutdown_connections 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- 172.21.15.145:0/3546348474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c1033d0 0x7f821c109c60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- 172.21.15.145:0/3546348474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c102a10 0x7f821c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- 172.21.15.145:0/3546348474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c0ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- 172.21.15.145:0/3546348474 >> 172.21.15.145:0/3546348474 conn(0x7f821c0fb2b0 msgr2=0x7f821c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- 172.21.15.145:0/3546348474 shutdown_connections 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- 172.21.15.145:0/3546348474 wait complete. 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 Processor -- start 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- start start 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:07.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c1033d0 0x7f821c100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c10d380 con 0x7f821c102a10 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.166+0000 7f82246b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c10d200 con 0x7f821c0ff9a0 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82246b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c10d500 con 0x7f821c1033d0 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37044/0 (socket says 172.21.15.145:37044) 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 -- 172.21.15.145:0/564300891 learned_addr learned my addr 172.21.15.145:0/564300891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8222c4f640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c1033d0 0x7f821c100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f822244e640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 -- 172.21.15.145:0/564300891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c1033d0 msgr2=0x7f821c100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c1033d0 0x7f821c100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 -- 172.21.15.145:0/564300891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 msgr2=0x7f821c102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 -- 172.21.15.145:0/564300891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c101790 con 0x7f821c102a10 2024-03-28T18:36:07.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f822244e640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f8221c4d640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f821800e9f0 tx=0x7f821800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800cd80 con 0x7f821c102a10 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8218005500 con 0x7f821c102a10 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c1a1ae0 con 0x7f821c102a10 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8218010690 con 0x7f821c102a10 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.167+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1a1fe0 con 0x7f821c102a10 2024-03-28T18:36:07.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.168+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c108b40 con 0x7f821c102a10 2024-03-28T18:36:07.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.169+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f82180046c0 con 0x7f821c102a10 2024-03-28T18:36:07.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.170+0000 7f82137fe640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 0x7f82080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:07.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.170+0000 7f822244e640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 0x7f82080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:07.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.170+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f82180cf3b0 con 0x7f821c102a10 2024-03-28T18:36:07.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.170+0000 7f822244e640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 0x7f82080a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f821c069a30 tx=0x7f820c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:07.174 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.174+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8218053740 con 0x7f821c102a10 2024-03-28T18:36:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:07.388+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f821c02d170 con 0x7f821c102a10 2024-03-28T18:36:07.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:07 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1939556641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:07.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:07 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/844703002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:07.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:07 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/237008476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:07 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1939556641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:07 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/844703002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:07 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/237008476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:07.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:07 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1939556641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:36:07.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:07 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/844703002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:07.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:07 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/237008476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:08.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.060+0000 7f82137fe640 1 -- 172.21.15.145:0/564300891 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f821809c210 con 0x7f821c102a10 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 msgr2=0x7f82080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 0x7f82080a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f821c069a30 tx=0x7f820c005d30 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 msgr2=0x7f821c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f821800e9f0 tx=0x7f821800edf0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 shutdown_connections 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.062+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f82080a2800 0x7f82080a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f821c1033d0 0x7f821c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f821c102a10 0x7f821c1009c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 --2- 172.21.15.145:0/564300891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f821c0ff9a0 0x7f821c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 >> 172.21.15.145:0/564300891 conn(0x7f821c0fb2b0 msgr2=0x7f821c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 shutdown_connections 2024-03-28T18:36:08.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.063+0000 7f82246b2640 1 -- 172.21.15.145:0/564300891 wait complete. 2024-03-28T18:36:08.075 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-28T18:36:08.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 -- 172.21.15.145:0/423183631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9401005a0 msgr2=0x7fc9401009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 --2- 172.21.15.145:0/423183631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9401005a0 0x7fc9401009c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc938009900 tx=0x7fc93802bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 -- 172.21.15.145:0/423183631 shutdown_connections 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 --2- 172.21.15.145:0/423183631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc940102150 0x7fc9401089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 --2- 172.21.15.145:0/423183631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940101790 0x7fc940101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 --2- 172.21.15.145:0/423183631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9401005a0 0x7fc9401009c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 -- 172.21.15.145:0/423183631 >> 172.21.15.145:0/423183631 conn(0x7fc9400fbb30 msgr2=0x7fc9400fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 -- 172.21.15.145:0/423183631 shutdown_connections 2024-03-28T18:36:08.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 -- 172.21.15.145:0/423183631 wait complete. 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.181+0000 7fc9448a1640 1 Processor -- start 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 -- start start 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9401005a0 0x7fc9401979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940102150 0x7fc94019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94010c230 con 0x7fc940101790 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94010c0b0 con 0x7fc9401005a0 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94010c3b0 con 0x7fc940102150 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37070/0 (socket says 172.21.15.145:37070) 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 -- 172.21.15.145:0/2300797760 learned_addr learned my addr 172.21.15.145:0/2300797760 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93ed9d640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940102150 0x7fc94019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:08.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93e59c640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9401005a0 0x7fc9401979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 -- 172.21.15.145:0/2300797760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940102150 msgr2=0x7fc94019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940102150 0x7fc94019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 -- 172.21.15.145:0/2300797760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9401005a0 msgr2=0x7fc9401979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9401005a0 0x7fc9401979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 -- 172.21.15.145:0/2300797760 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94019f5c0 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc93dd9b640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc93400eb90 tx=0x7fc93400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934004d20 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc93400ee90 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94019f8b0 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.182+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934010830 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.183+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94019fe00 con 0x7fc940101790 2024-03-28T18:36:08.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.183+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9401a0030 con 0x7fc940101790 2024-03-28T18:36:08.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.185+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc934005020 con 0x7fc940101790 2024-03-28T18:36:08.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.185+0000 7fc92f7fe640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 0x7fc9280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:08.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.185+0000 7fc93e59c640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 0x7fc9280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:08.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.185+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc9340cf220 con 0x7fc940101790 2024-03-28T18:36:08.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.185+0000 7fc93e59c640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 0x7fc9280a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc938000c00 tx=0x7fc938005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:08.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.190+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc93409c080 con 0x7fc940101790 2024-03-28T18:36:08.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:08.406+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fc9401a02a0 con 0x7fc940101790 2024-03-28T18:36:08.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:08 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:36:08.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:08 smithi192 ceph-mon[36589]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2024-03-28T18:36:08.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:08 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:36:08.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:08 smithi164 ceph-mon[36248]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2024-03-28T18:36:08.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:08 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:36:08.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:08 smithi145 ceph-mon[32151]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2024-03-28T18:36:09.067 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.067+0000 7fc92f7fe640 1 -- 172.21.15.145:0/2300797760 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc93409b980 con 0x7fc940101790 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 msgr2=0x7fc9280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 0x7fc9280a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc938000c00 tx=0x7fc938005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 msgr2=0x7fc940197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc93400eb90 tx=0x7fc93400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 shutdown_connections 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9280a2880 0x7fc9280a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc940102150 0x7fc94019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc940101790 0x7fc940197ee0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 --2- 172.21.15.145:0/2300797760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9401005a0 0x7fc9401979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 >> 172.21.15.145:0/2300797760 conn(0x7fc9400fbb30 msgr2=0x7fc9400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 shutdown_connections 2024-03-28T18:36:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.070+0000 7fc9448a1640 1 -- 172.21.15.145:0/2300797760 wait complete. 2024-03-28T18:36:09.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T18:36:09.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 -- 172.21.15.145:0/2989208800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af4102150 msgr2=0x7f1af41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:09.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2989208800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af4102150 0x7f1af41089e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009900 tx=0x7f1aec02bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:09.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 -- 172.21.15.145:0/2989208800 shutdown_connections 2024-03-28T18:36:09.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2989208800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af4102150 0x7f1af41089e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2989208800 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4101790 0x7f1af4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2989208800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af41005a0 0x7f1af41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 -- 172.21.15.145:0/2989208800 >> 172.21.15.145:0/2989208800 conn(0x7f1af40fbb10 msgr2=0x7f1af40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 -- 172.21.15.145:0/2989208800 shutdown_connections 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.190+0000 7f1afb51b640 1 -- 172.21.15.145:0/2989208800 wait complete. 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 Processor -- start 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 -- start start 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af4101790 0x7f1af419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4102150 0x7f1af419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410c230 con 0x7f1af41005a0 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410c0b0 con 0x7f1af4102150 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1afb51b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410c3b0 con 0x7f1af4101790 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1af92b7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1af92b7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37100/0 (socket says 172.21.15.145:37100) 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1af92b7640 1 -- 172.21.15.145:0/2358256617 learned_addr learned my addr 172.21.15.145:0/2358256617 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:09.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.191+0000 7f1af8ab6640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af4101790 0x7f1af419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af9ab8640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4102150 0x7f1af419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 -- 172.21.15.145:0/2358256617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af4101790 msgr2=0x7f1af419c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af4101790 0x7f1af419c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 -- 172.21.15.145:0/2358256617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4102150 msgr2=0x7f1af419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4102150 0x7f1af419ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 -- 172.21.15.145:0/2358256617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af419f550 con 0x7f1af41005a0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1af92b7640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1af000eb70 tx=0x7f1af000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0003b70 con 0x7f1af41005a0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1af000ee70 con 0x7f1af41005a0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af419f840 con 0x7f1af41005a0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0010830 con 0x7f1af41005a0 2024-03-28T18:36:09.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.192+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af419fd90 con 0x7f1af41005a0 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.193+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af41078c0 con 0x7f1af41005a0 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.194+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1af0004020 con 0x7f1af41005a0 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.194+0000 7f1aea7fc640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 0x7f1ae00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.195+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1af00cf340 con 0x7f1af41005a0 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.195+0000 7f1af8ab6640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 0x7f1ae00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:09.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.195+0000 7f1af8ab6640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 0x7f1ae00a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1af4069140 tx=0x7f1ae400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:09.198 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.199+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1af009c1a0 con 0x7f1af41005a0 2024-03-28T18:36:09.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:09 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:36:09.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:09 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:09.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:09 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:36:09.410 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:09.410+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f1af40622d0 con 0x7f1af41005a0 2024-03-28T18:36:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:09 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:36:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:09 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:09 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:36:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/564300891' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:36:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:09 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:36:10.075 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.076+0000 7f1aea7fc640 1 -- 172.21.15.145:0/2358256617 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1af009baa0 con 0x7f1af41005a0 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 msgr2=0x7f1ae00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 0x7f1ae00a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1af4069140 tx=0x7f1ae400a2d0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 msgr2=0x7f1af40693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1af000eb70 tx=0x7f1af000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 shutdown_connections 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1af4102150 0x7f1af419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ae00a2880 0x7f1ae00a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1af4101790 0x7f1af419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 --2- 172.21.15.145:0/2358256617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1af41005a0 0x7f1af40693d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 >> 172.21.15.145:0/2358256617 conn(0x7f1af40fbb10 msgr2=0x7f1af4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 shutdown_connections 2024-03-28T18:36:10.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.079+0000 7f1afb51b640 1 -- 172.21.15.145:0/2358256617 wait complete. 2024-03-28T18:36:10.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 -- 172.21.15.145:0/2026543657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c1005c0 msgr2=0x7f9b8c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 --2- 172.21.15.145:0/2026543657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c1009e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009900 tx=0x7f9b8802bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 -- 172.21.15.145:0/2026543657 shutdown_connections 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 --2- 172.21.15.145:0/2026543657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c102170 0x7f9b8c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 --2- 172.21.15.145:0/2026543657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 0x7f9b8c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 --2- 172.21.15.145:0/2026543657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c1009e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 -- 172.21.15.145:0/2026543657 >> 172.21.15.145:0/2026543657 conn(0x7f9b8c0fbb50 msgr2=0x7f9b8c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 -- 172.21.15.145:0/2026543657 shutdown_connections 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.200+0000 7f9b9395b640 1 -- 172.21.15.145:0/2026543657 wait complete. 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 Processor -- start 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 -- start start 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 0x7f9b8c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:10.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c10c210 con 0x7f9b8c102170 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c10c090 con 0x7f9b8c1017b0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b9395b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c10c390 con 0x7f9b8c1005c0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b91ef8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b91ef8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37112/0 (socket says 172.21.15.145:37112) 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b91ef8640 1 -- 172.21.15.145:0/3437152051 learned_addr learned my addr 172.21.15.145:0/3437152051 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b916f7640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b90ef6640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 0x7f9b8c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b91ef8640 1 -- 172.21.15.145:0/3437152051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c1005c0 msgr2=0x7f9b8c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.201+0000 7f9b91ef8640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b91ef8640 1 -- 172.21.15.145:0/3437152051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 msgr2=0x7f9b8c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b91ef8640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 0x7f9b8c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b91ef8640 1 -- 172.21.15.145:0/3437152051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c19f650 con 0x7f9b8c102170 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b91ef8640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400eb10 tx=0x7f9b8400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84004b20 con 0x7f9b8c102170 2024-03-28T18:36:10.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b84004cc0 con 0x7f9b8c102170 2024-03-28T18:36:10.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c19f940 con 0x7f9b8c102170 2024-03-28T18:36:10.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84010760 con 0x7f9b8c102170 2024-03-28T18:36:10.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.202+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c19fe40 con 0x7f9b8c102170 2024-03-28T18:36:10.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.203+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c1a0130 con 0x7f9b8c102170 2024-03-28T18:36:10.205 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.204+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9b84005020 con 0x7f9b8c102170 2024-03-28T18:36:10.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.204+0000 7f9b827fc640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 0x7f9b780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:10.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.205+0000 7f9b916f7640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 0x7f9b780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:10.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.205+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9b840cf210 con 0x7f9b8c102170 2024-03-28T18:36:10.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.205+0000 7f9b916f7640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 0x7f9b780a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9b88000c00 tx=0x7f9b88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:10.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.209+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b8409c070 con 0x7f9b8c102170 2024-03-28T18:36:10.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:36:10.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-mon[36589]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T18:36:10.422 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:10.422+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f9b8c1a0520 con 0x7f9b8c102170 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:36:10.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:10 smithi164 ceph-mon[36248]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T18:36:10.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:36:10.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2300797760' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:36:10.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:36:10.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T18:36:10.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-28T18:36:10.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-28T18:36:10.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:36:10.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:10 smithi145 ceph-mon[32151]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T18:36:10.906 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:36:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:36:10.554+0000 7f84ef9a3700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T18:36:11.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.081+0000 7f9b827fc640 1 -- 172.21.15.145:0/3437152051 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9b8409b970 con 0x7f9b8c102170 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 msgr2=0x7f9b780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 0x7f9b780a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9b88000c00 tx=0x7f9b88005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 msgr2=0x7f9b8c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400eb10 tx=0x7f9b8400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 shutdown_connections 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b8c102170 0x7f9b8c19efe0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b8c1017b0 0x7f9b8c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9b780a2800 0x7f9b780a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 --2- 172.21.15.145:0/3437152051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b8c1005c0 0x7f9b8c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 >> 172.21.15.145:0/3437152051 conn(0x7f9b8c0fbb50 msgr2=0x7f9b8c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:11.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 shutdown_connections 2024-03-28T18:36:11.085 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.084+0000 7f9b9395b640 1 -- 172.21.15.145:0/3437152051 wait complete. 2024-03-28T18:36:11.099 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-28T18:36:11.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 -- 172.21.15.145:0/3347791989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76740ff980 msgr2=0x7f76740ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 --2- 172.21.15.145:0/3347791989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76740ff980 0x7f76740ffda0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7664009cd0 tx=0x7f766402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 -- 172.21.15.145:0/3347791989 shutdown_connections 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 --2- 172.21.15.145:0/3347791989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674103310 0x7f7674109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 --2- 172.21.15.145:0/3347791989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7674102950 0x7f7674102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 --2- 172.21.15.145:0/3347791989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76740ff980 0x7f76740ffda0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 -- 172.21.15.145:0/3347791989 >> 172.21.15.145:0/3347791989 conn(0x7f76740fb290 msgr2=0x7f76740fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 -- 172.21.15.145:0/3347791989 shutdown_connections 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.206+0000 7f7679b22640 1 -- 172.21.15.145:0/3347791989 wait complete. 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 Processor -- start 2024-03-28T18:36:11.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 -- start start 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 0x7f76741979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674102950 0x7f7674197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767410d330 con 0x7f7674103310 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767410d1b0 con 0x7f76740ff980 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7679b22640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767410d4b0 con 0x7f7674102950 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7673fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f76737fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 0x7f76741979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:11.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7673fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37124/0 (socket says 172.21.15.145:37124) 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7673fff640 1 -- 172.21.15.145:0/956604237 learned_addr learned my addr 172.21.15.145:0/956604237 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.207+0000 7f7672ffd640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674102950 0x7f7674197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 -- 172.21.15.145:0/956604237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674102950 msgr2=0x7f7674197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674102950 0x7f7674197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 -- 172.21.15.145:0/956604237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 msgr2=0x7f76741979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 0x7f76741979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 -- 172.21.15.145:0/956604237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767419f5f0 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f76737fe640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 0x7f76741979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7673fff640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f766c00d750 tx=0x7f766c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c017070 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f766c005500 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767419f8e0 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c004d90 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.208+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767419fde0 con 0x7f7674103310 2024-03-28T18:36:11.210 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.209+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674108b00 con 0x7f7674103310 2024-03-28T18:36:11.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.210+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f766c004f30 con 0x7f7674103310 2024-03-28T18:36:11.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.210+0000 7f7670ff9640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 0x7f765c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:11.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.211+0000 7f76737fe640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 0x7f765c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:11.211 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.211+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f766c0ce5d0 con 0x7f7674103310 2024-03-28T18:36:11.212 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.211+0000 7f76737fe640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 0x7f765c0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7664009bf0 tx=0x7f76640325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:11.214 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.215+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f766c09ade0 con 0x7f7674103310 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:active 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:36:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:36:11.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:36:11.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:active 2024-03-28T18:36:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:36:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:36:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:36:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:36:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:36:11.416 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:11.416+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f7674061ae0 con 0x7f765c0a2800 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2358256617' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] up:active 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:36:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:36:11.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:36:11.421+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:36:11.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:36:11.421+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:36:11.776 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:36:11.421+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:36:11.777 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:36:11.421+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:36:11.777 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:36:11 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:36:11.421+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:36:12.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.032+0000 7f7670ff9640 1 -- 172.21.15.145:0/956604237 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7674061ae0 con 0x7f765c0a2800 2024-03-28T18:36:12.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 msgr2=0x7f765c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 0x7f765c0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7664009bf0 tx=0x7f76640325b0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 msgr2=0x7f767419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f766c00d750 tx=0x7f766c00dc10 comp rx=0 tx=0).stop 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 shutdown_connections 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7674103310 0x7f767419ef80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f765c0a2800 0x7f765c0a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7674102950 0x7f7674197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 --2- 172.21.15.145:0/956604237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76740ff980 0x7f76741979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 >> 172.21.15.145:0/956604237 conn(0x7f76740fb290 msgr2=0x7f7674105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 shutdown_connections 2024-03-28T18:36:12.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.034+0000 7f7679b22640 1 -- 172.21.15.145:0/956604237 wait complete. 2024-03-28T18:36:12.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3381250258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac100f30 msgr2=0x7ff6ac1013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3381250258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac100f30 0x7ff6ac1013b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009900 tx=0x7ff69c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3381250258 shutdown_connections 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3381250258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3381250258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac100f30 0x7ff6ac1013b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3381250258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3381250258 >> 172.21.15.145:0/3381250258 conn(0x7ff6ac0fb2d0 msgr2=0x7ff6ac0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3381250258 shutdown_connections 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3381250258 wait complete. 2024-03-28T18:36:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 Processor -- start 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- start start 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac1096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac10c220 con 0x7ff6ac1018f0 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac10c0a0 con 0x7ff6ac100f30 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.154+0000 7ff6b42ac640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac10c3a0 con 0x7ff6ac0ffd40 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2849640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2048640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac1096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2048640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac1096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:47302/0 (socket says 172.21.15.145:47302) 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:58776/0 (socket says 172.21.15.145:58776) 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2048640 1 -- 172.21.15.145:0/3351443693 learned_addr learned my addr 172.21.15.145:0/3351443693 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:12.157 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2849640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37160/0 (socket says 172.21.15.145:37160) 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 -- 172.21.15.145:0/3351443693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 msgr2=0x7ff6ac1096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac1096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 -- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 msgr2=0x7ff6ac19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 -- 172.21.15.145:0/3351443693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac19f670 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b2849640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b1847640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009230 tx=0x7ff69c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c041070 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69c004450 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac19f8a0 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c003d80 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.155+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ac19fd90 con 0x7ff6ac100f30 2024-03-28T18:36:12.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.156+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ac1070e0 con 0x7ff6ac100f30 2024-03-28T18:36:12.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.157+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff69c044070 con 0x7ff6ac100f30 2024-03-28T18:36:12.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.158+0000 7ff6a37fe640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 0x7ff6980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.158+0000 7ff6b2048640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 0x7ff6980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.158+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff69c0f0680 con 0x7ff6ac100f30 2024-03-28T18:36:12.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.158+0000 7ff6b2048640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 0x7ff6980a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40058c0 tx=0x7ff6a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.165+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff69c0bd4e0 con 0x7ff6ac100f30 2024-03-28T18:36:12.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:active 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:active 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:36:12.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T18:36:12.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T18:36:12.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:12 smithi164 ceph-mon[36248]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T18:36:12.366 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.367+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6ac061b70 con 0x7ff6980a2800 2024-03-28T18:36:12.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.391+0000 7ff6a37fe640 1 -- 172.21.15.145:0/3351443693 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6ac061b70 con 0x7ff6980a2800 2024-03-28T18:36:12.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 msgr2=0x7ff6980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 0x7ff6980a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40058c0 tx=0x7ff6a4005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 msgr2=0x7ff6ac19c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009230 tx=0x7ff69c02c650 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 shutdown_connections 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6ac1018f0 0x7ff6ac19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6ac100f30 0x7ff6ac19c740 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff6980a2800 0x7ff6980a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 --2- 172.21.15.145:0/3351443693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ac0ffd40 0x7ff6ac1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.393+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 >> 172.21.15.145:0/3351443693 conn(0x7ff6ac0fb2d0 msgr2=0x7ff6ac1042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.394+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 shutdown_connections 2024-03-28T18:36:12.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.394+0000 7ff6b42ac640 1 -- 172.21.15.145:0/3351443693 wait complete. 2024-03-28T18:36:12.403 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:36:12.403 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:active 2024-03-28T18:36:12.403 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:active 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T18:36:12.404 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:12 smithi145 ceph-mon[32151]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T18:36:12.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T18:36:12.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3437152051' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] up:active 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] up:active 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T18:36:12.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:12 smithi192 ceph-mon[36589]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 -- 172.21.15.145:0/2906315138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee81021f0 msgr2=0x7f1ee8108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 --2- 172.21.15.145:0/2906315138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee81021f0 0x7f1ee8108a80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009ca0 tx=0x7f1ee002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 -- 172.21.15.145:0/2906315138 shutdown_connections 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 --2- 172.21.15.145:0/2906315138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee81021f0 0x7f1ee8108a80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 --2- 172.21.15.145:0/2906315138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee8101830 0x7f1ee8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 --2- 172.21.15.145:0/2906315138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee8100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 -- 172.21.15.145:0/2906315138 >> 172.21.15.145:0/2906315138 conn(0x7f1ee80fbbb0 msgr2=0x7f1ee80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 -- 172.21.15.145:0/2906315138 shutdown_connections 2024-03-28T18:36:12.512 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.512+0000 7f1eecff3640 1 -- 172.21.15.145:0/2906315138 wait complete. 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 Processor -- start 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 -- start start 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee81968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee81021f0 0x7f1ee819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810c2a0 con 0x7f1ee8101830 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810c120 con 0x7f1ee8100640 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1eecff3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee810c420 con 0x7f1ee81021f0 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1edffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee81968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1edffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58040/0 (socket says 172.21.15.145:58040) 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1edffff640 1 -- 172.21.15.145:0/3683279187 learned_addr learned my addr 172.21.15.145:0/3683279187 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee6d9d640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee81021f0 0x7f1ee819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 -- 172.21.15.145:0/3683279187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee81021f0 msgr2=0x7f1ee819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee81021f0 0x7f1ee819de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 -- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 msgr2=0x7f1ee8196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 -- 172.21.15.145:0/3683279187 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee819e4e0 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1edffff640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.513+0000 7f1ee659c640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee81968c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800e9c0 tx=0x7f1ed800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.514+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800ccc0 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.514+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee819e7d0 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.514+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ed80035f0 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.514+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800ce60 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.514+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee8069f20 con 0x7f1ee8100640 2024-03-28T18:36:12.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.515+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee806a9a0 con 0x7f1ee8100640 2024-03-28T18:36:12.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.516+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1ed80037a0 con 0x7f1ee8100640 2024-03-28T18:36:12.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.516+0000 7f1edf7fe640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 0x7f1ed40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.516+0000 7f1edffff640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 0x7f1ed40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.516+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1ed80cf0c0 con 0x7f1ee8100640 2024-03-28T18:36:12.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.517+0000 7f1edffff640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 0x7f1ed40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8197dd0 tx=0x7f1ed0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.521+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ed809bf20 con 0x7f1ee8100640 2024-03-28T18:36:12.724 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.725+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee8107790 con 0x7f1ed40a2800 2024-03-28T18:36:12.730 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.730+0000 7f1edf7fe640 1 -- 172.21.15.145:0/3683279187 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f1ee8107790 con 0x7f1ed40a2800 2024-03-28T18:36:12.730 INFO:teuthology.orchestra.run.smithi145.stdout:/volumes/qa/sv_0/f3fa7b4a-85b8-4356-b057-611e6b44cb43 2024-03-28T18:36:12.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 msgr2=0x7f1ed40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 0x7f1ed40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8197dd0 tx=0x7f1ed0005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 msgr2=0x7f1ee81968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee81968c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800e9c0 tx=0x7f1ed800ee80 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 shutdown_connections 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1ed40a2800 0x7f1ed40a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ee81021f0 0x7f1ee819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ee8101830 0x7f1ee8196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 --2- 172.21.15.145:0/3683279187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ee8100640 0x7f1ee81968c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.732+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 >> 172.21.15.145:0/3683279187 conn(0x7f1ee80fbbb0 msgr2=0x7f1ee8104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.733+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 shutdown_connections 2024-03-28T18:36:12.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.733+0000 7f1eecff3640 1 -- 172.21.15.145:0/3683279187 wait complete. 2024-03-28T18:36:12.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 -- 172.21.15.145:0/275263383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2681017b0 msgr2=0x7fc268101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/275263383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2681017b0 0x7fc268101c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc250009900 tx=0x7fc25002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 -- 172.21.15.145:0/275263383 shutdown_connections 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/275263383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc268102170 0x7fc268108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/275263383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2681017b0 0x7fc268101c30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/275263383 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681005c0 0x7fc2681009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 -- 172.21.15.145:0/275263383 >> 172.21.15.145:0/275263383 conn(0x7fc2680fbb50 msgr2=0x7fc2680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 -- 172.21.15.145:0/275263383 shutdown_connections 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 -- 172.21.15.145:0/275263383 wait complete. 2024-03-28T18:36:12.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.854+0000 7fc26d8f8640 1 Processor -- start 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 -- start start 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 0x7fc26806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 0x7fc26819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc268102170 0x7fc26819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810c240 con 0x7fc2681017b0 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810c0c0 con 0x7fc2681005c0 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26810c3c0 con 0x7fc268102170 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc266ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 0x7fc26806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 0x7fc26819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc266ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 0x7fc26806a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:40664/0 (socket says 172.21.15.145:40664) 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc266ffd640 1 -- 172.21.15.145:0/1479614495 learned_addr learned my addr 172.21.15.145:0/1479614495 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc2677fe640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc268102170 0x7fc26819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 -- 172.21.15.145:0/1479614495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc268102170 msgr2=0x7fc26819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc268102170 0x7fc26819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 -- 172.21.15.145:0/1479614495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 msgr2=0x7fc26806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 0x7fc26806a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 -- 172.21.15.145:0/1479614495 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26819f660 con 0x7fc2681017b0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc25ffff640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 0x7fc26819c730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc250009230 tx=0x7fc25002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250041070 con 0x7fc2681017b0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26819f890 con 0x7fc2681017b0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc250004450 con 0x7fc2681017b0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.855+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250003df0 con 0x7fc2681017b0 2024-03-28T18:36:12.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.856+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26819fd80 con 0x7fc2681017b0 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.857+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2681078e0 con 0x7fc2681017b0 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.857+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc250044070 con 0x7fc2681017b0 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.858+0000 7fc26c8f6640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 0x7fc2540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.858+0000 7fc266ffd640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 0x7fc2540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.858+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc2500f06c0 con 0x7fc2681017b0 2024-03-28T18:36:12.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.858+0000 7fc266ffd640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 0x7fc2540a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc2580095f0 tx=0x7fc258005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:12.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:12.862+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc2500bced0 con 0x7fc2681017b0 2024-03-28T18:36:13.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.019+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc2500035f0 con 0x7fc2681017b0 2024-03-28T18:36:13.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.102+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fc268061b70 con 0x7fc2540a2800 2024-03-28T18:36:13.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.125+0000 7fc26c8f6640 1 -- 172.21.15.145:0/1479614495 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc268061b70 con 0x7fc2540a2800 2024-03-28T18:36:13.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 msgr2=0x7fc2540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 0x7fc2540a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc2580095f0 tx=0x7fc258005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 msgr2=0x7fc26819c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 0x7fc26819c730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc250009230 tx=0x7fc25002c650 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 shutdown_connections 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2540a2800 0x7fc2540a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc268102170 0x7fc26819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2681017b0 0x7fc26819c730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 --2- 172.21.15.145:0/1479614495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2681005c0 0x7fc26806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.127+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 >> 172.21.15.145:0/1479614495 conn(0x7fc2680fbb50 msgr2=0x7fc268104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.128+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 shutdown_connections 2024-03-28T18:36:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.128+0000 7fc26d8f8640 1 -- 172.21.15.145:0/1479614495 wait complete. 2024-03-28T18:36:13.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 -- 172.21.15.145:0/3159099548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c1011f0 msgr2=0x7f1a6c108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 --2- 172.21.15.145:0/3159099548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c108a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a68009ca0 tx=0x7f1a6802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 -- 172.21.15.145:0/3159099548 shutdown_connections 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 --2- 172.21.15.145:0/3159099548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c108a90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 --2- 172.21.15.145:0/3159099548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c100830 0x7f1a6c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 --2- 172.21.15.145:0/3159099548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 0x7f1a6c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 -- 172.21.15.145:0/3159099548 >> 172.21.15.145:0/3159099548 conn(0x7f1a6c0fb2a0 msgr2=0x7f1a6c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 -- 172.21.15.145:0/3159099548 shutdown_connections 2024-03-28T18:36:13.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.248+0000 7f1a73f89640 1 -- 172.21.15.145:0/3159099548 wait complete. 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 Processor -- start 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 -- start start 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 0x7f1a6c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10c270 con 0x7f1a6c100830 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10c0f0 con 0x7f1a6c1011f0 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a73f89640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10c3f0 con 0x7f1a6c101be0 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58074/0 (socket says 172.21.15.145:58074) 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 -- 172.21.15.145:0/1697161807 learned_addr learned my addr 172.21.15.145:0/1697161807 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71524640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.251 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a72526640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 0x7f1a6c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 -- 172.21.15.145:0/1697161807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 msgr2=0x7f1a6c19de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 0x7f1a6c19de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 -- 172.21.15.145:0/1697161807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c1011f0 msgr2=0x7f1a6c196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 -- 172.21.15.145:0/1697161807 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c19e480 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.249+0000 7f1a71d25640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400e9c0 tx=0x7f1a6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.250+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400cd80 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.250+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a640035f0 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.250+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c19e770 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.250+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a64010690 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.250+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c19ec70 con 0x7f1a6c100830 2024-03-28T18:36:13.252 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.251+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c107970 con 0x7f1a6c100830 2024-03-28T18:36:13.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.252+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1a640037a0 con 0x7f1a6c100830 2024-03-28T18:36:13.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.252+0000 7f1a62ffd640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 0x7f1a580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.252+0000 7f1a71524640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 0x7f1a580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.252+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1a640cf260 con 0x7f1a6c100830 2024-03-28T18:36:13.253 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.253+0000 7f1a71524640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 0x7f1a580a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c197d70 tx=0x7f1a5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.257+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a6409bf60 con 0x7f1a6c100830 2024-03-28T18:36:13.406 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:13 smithi145 ceph-mon[32151]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:13.406 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:13 smithi145 ceph-mon[32151]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T18:36:13.406 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:13 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:active 2024-03-28T18:36:13.406 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:13 smithi145 ceph-mon[32151]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T18:36:13.406 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:13 smithi145 ceph-mon[32151]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:36:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:13 smithi192 ceph-mon[36589]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:13 smithi192 ceph-mon[36589]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T18:36:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:13 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:active 2024-03-28T18:36:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:13 smithi192 ceph-mon[36589]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T18:36:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:13 smithi192 ceph-mon[36589]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:36:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:13 smithi164 ceph-mon[36248]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:13 smithi164 ceph-mon[36248]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T18:36:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:13 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] up:active 2024-03-28T18:36:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:13 smithi164 ceph-mon[36248]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T18:36:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:13 smithi164 ceph-mon[36248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:36:13.486 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.486+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a6c061ae0 con 0x7f1a580a2890 2024-03-28T18:36:13.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.491+0000 7f1a62ffd640 1 -- 172.21.15.145:0/1697161807 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f1a6c061ae0 con 0x7f1a580a2890 2024-03-28T18:36:13.491 INFO:teuthology.orchestra.run.smithi145.stdout:/volumes/qa/sv_1/76215e0f-6dde-4a84-9c7d-3d313c968b82 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 msgr2=0x7f1a580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 0x7f1a580a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c197d70 tx=0x7f1a5c005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 msgr2=0x7f1a6c196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400e9c0 tx=0x7f1a6400ee80 comp rx=0 tx=0).stop 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 shutdown_connections 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1a580a2890 0x7f1a580a4d50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a6c101be0 0x7f1a6c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a6c1011f0 0x7f1a6c196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 --2- 172.21.15.145:0/1697161807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a6c100830 0x7f1a6c196860 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 >> 172.21.15.145:0/1697161807 conn(0x7f1a6c0fb2a0 msgr2=0x7f1a6c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 shutdown_connections 2024-03-28T18:36:13.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.493+0000 7f1a73f89640 1 -- 172.21.15.145:0/1697161807 wait complete. 2024-03-28T18:36:13.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:36:13.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3361942663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 msgr2=0x7f8f48102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3361942663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48102d60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009900 tx=0x7f8f3402bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:13.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3361942663 shutdown_connections 2024-03-28T18:36:13.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3361942663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481032a0 0x7f8f48109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3361942663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48102d60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3361942663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481016f0 0x7f8f48101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3361942663 >> 172.21.15.145:0/3361942663 conn(0x7f8f480fcc40 msgr2=0x7f8f480ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3361942663 shutdown_connections 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.608+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3361942663 wait complete. 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 Processor -- start 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 -- start start 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481016f0 0x7f8f481979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481032a0 0x7f8f4819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4810d370 con 0x7f8f481028e0 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4810d1f0 con 0x7f8f481032a0 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ef59640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4810d4f0 con 0x7f8f481016f0 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58090/0 (socket says 172.21.15.145:58090) 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 -- 172.21.15.145:0/3218795608 learned_addr learned my addr 172.21.15.145:0/3218795608 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4ccf5640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481016f0 0x7f8f481979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f4d4f6640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481032a0 0x7f8f4819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 -- 172.21.15.145:0/3218795608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481016f0 msgr2=0x7f8f481979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481016f0 0x7f8f481979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 -- 172.21.15.145:0/3218795608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481032a0 msgr2=0x7f8f4819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481032a0 0x7f8f4819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.609+0000 7f8f47fff640 1 -- 172.21.15.145:0/3218795608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4819f5e0 con 0x7f8f481028e0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f47fff640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009230 tx=0x7f8f3402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34041070 con 0x7f8f481028e0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f34004450 con 0x7f8f481028e0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34003e20 con 0x7f8f481028e0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4819f870 con 0x7f8f481028e0 2024-03-28T18:36:13.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.610+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4819fd60 con 0x7f8f481028e0 2024-03-28T18:36:13.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.611+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48108a10 con 0x7f8f481028e0 2024-03-28T18:36:13.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.612+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8f34044070 con 0x7f8f481028e0 2024-03-28T18:36:13.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.613+0000 7f8f45ffb640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 0x7f8f380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.613+0000 7f8f4ccf5640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 0x7f8f380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.613+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8f340f0c20 con 0x7f8f481028e0 2024-03-28T18:36:13.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.614+0000 7f8f4ccf5640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 0x7f8f380a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f48102740 tx=0x7f8f3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.617+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f340bd3e0 con 0x7f8f481028e0 2024-03-28T18:36:13.849 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.849+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f4802d070 con 0x7f8f481028e0 2024-03-28T18:36:13.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.850+0000 7f8f45ffb640 1 -- 172.21.15.145:0/3218795608 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7f8f340c1480 con 0x7f8f481028e0 2024-03-28T18:36:13.850 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:13.850 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:36:13.851 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:36:13.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 msgr2=0x7f8f380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 0x7f8f380a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f48102740 tx=0x7f8f3c005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 msgr2=0x7f8f48197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009230 tx=0x7f8f3402c650 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 shutdown_connections 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f481032a0 0x7f8f4819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f481028e0 0x7f8f48197f00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8f380a2890 0x7f8f380a4d50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 --2- 172.21.15.145:0/3218795608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f481016f0 0x7f8f481979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.852+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 >> 172.21.15.145:0/3218795608 conn(0x7f8f480fcc40 msgr2=0x7f8f48105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.853+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 shutdown_connections 2024-03-28T18:36:13.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.853+0000 7f8f4ef59640 1 -- 172.21.15.145:0/3218795608 wait complete. 2024-03-28T18:36:13.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:36:13.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 -- 172.21.15.145:0/3096615422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41005c0 msgr2=0x7f59e41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 --2- 172.21.15.145:0/3096615422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41005c0 0x7f59e41009e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009900 tx=0x7f59d402bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:13.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 -- 172.21.15.145:0/3096615422 shutdown_connections 2024-03-28T18:36:13.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 --2- 172.21.15.145:0/3096615422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 0x7f59e4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 --2- 172.21.15.145:0/3096615422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41017b0 0x7f59e4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 --2- 172.21.15.145:0/3096615422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41005c0 0x7f59e41009e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 -- 172.21.15.145:0/3096615422 >> 172.21.15.145:0/3096615422 conn(0x7f59e40fbb30 msgr2=0x7f59e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:13.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 -- 172.21.15.145:0/3096615422 shutdown_connections 2024-03-28T18:36:13.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.972+0000 7f59e8897640 1 -- 172.21.15.145:0/3096615422 wait complete. 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 Processor -- start 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 -- start start 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41005c0 0x7f59e4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 0x7f59e41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.973+0000 7f59e8897640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e410c230 con 0x7f59e41017b0 2024-03-28T18:36:13.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e8897640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e410c0b0 con 0x7f59e4102170 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e8897640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e410c3b0 con 0x7f59e41005c0 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58094/0 (socket says 172.21.15.145:58094) 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 -- 172.21.15.145:0/687922659 learned_addr learned my addr 172.21.15.145:0/687922659 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41005c0 0x7f59e4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e2d9d640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 0x7f59e41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 -- 172.21.15.145:0/687922659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41005c0 msgr2=0x7f59e4199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41005c0 0x7f59e4199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 -- 172.21.15.145:0/687922659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 msgr2=0x7f59e41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 0x7f59e41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 -- 172.21.15.145:0/687922659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e41a1810 con 0x7f59e41017b0 2024-03-28T18:36:13.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59dbfff640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00e9f0 tx=0x7f59cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc00cd80 con 0x7f59e41017b0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59cc005500 con 0x7f59e41017b0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e41a1b00 con 0x7f59e41017b0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc010690 con 0x7f59e41017b0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.974+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e41a2000 con 0x7f59e41017b0 2024-03-28T18:36:13.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.975+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e41a2230 con 0x7f59e41017b0 2024-03-28T18:36:13.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.977+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f59cc0046c0 con 0x7f59e41017b0 2024-03-28T18:36:13.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.977+0000 7f59db7fe640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 0x7f59d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:13.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.978+0000 7f59e259c640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 0x7f59d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:13.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.978+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f59cc0cf400 con 0x7f59e41017b0 2024-03-28T18:36:13.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.978+0000 7f59e259c640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 0x7f59d00a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:13.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:13.981+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59cc053740 con 0x7f59e41017b0 2024-03-28T18:36:14.163 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.163+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59e41a2400 con 0x7f59e41017b0 2024-03-28T18:36:14.164 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.165+0000 7f59db7fe640 1 -- 172.21.15.145:0/687922659 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7f59cc09c100 con 0x7f59e41017b0 2024-03-28T18:36:14.165 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:14.165 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 msgr2=0x7f59d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 0x7f59d00a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d4005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 msgr2=0x7f59e419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00e9f0 tx=0x7f59cc00edf0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 shutdown_connections 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59e4102170 0x7f59e41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e41017b0 0x7f59e419a130 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f59d00a2890 0x7f59d00a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 --2- 172.21.15.145:0/687922659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e41005c0 0x7f59e4199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 >> 172.21.15.145:0/687922659 conn(0x7f59e40fbb30 msgr2=0x7f59e40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:14.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 shutdown_connections 2024-03-28T18:36:14.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.167+0000 7f59e8897640 1 -- 172.21.15.145:0/687922659 wait complete. 2024-03-28T18:36:14.177 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:14 smithi145 ceph-mon[32151]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.177 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:14 smithi145 ceph-mon[32151]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.177 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:14 smithi145 ceph-mon[32151]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T18:36:14.177 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:14 smithi145 ceph-mon[32151]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.177 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:14 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3218795608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:36:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T18:36:14.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:36:14.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/4099147207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb41021a0 msgr2=0x7f0eb4104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/4099147207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb41021a0 0x7f0eb4104590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009900 tx=0x7f0eb002bc30 comp rx=0 tx=0).stop 2024-03-28T18:36:14.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/4099147207 shutdown_connections 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/4099147207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb4107400 0x7f0eb41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/4099147207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/4099147207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb41021a0 0x7f0eb4104590 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/4099147207 >> 172.21.15.145:0/4099147207 conn(0x7f0eb40fbb30 msgr2=0x7f0eb40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.283+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/4099147207 shutdown_connections 2024-03-28T18:36:14.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/4099147207 wait complete. 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 Processor -- start 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 -- start start 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb41021a0 0x7f0eb41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4107400 0x7f0eb419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d340 con 0x7f0eb4104ad0 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d1c0 con 0x7f0eb4107400 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0ebbc3f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb410d4c0 con 0x7f0eb41021a0 2024-03-28T18:36:14.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0eb91da640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0eb91da640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58114/0 (socket says 172.21.15.145:58114) 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0eb91da640 1 -- 172.21.15.145:0/2249884848 learned_addr learned my addr 172.21.15.145:0/2249884848 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0eb99db640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb41021a0 0x7f0eb41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.284+0000 7f0eba1dc640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4107400 0x7f0eb419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 -- 172.21.15.145:0/2249884848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb41021a0 msgr2=0x7f0eb41979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb41021a0 0x7f0eb41979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 -- 172.21.15.145:0/2249884848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4107400 msgr2=0x7f0eb419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4107400 0x7f0eb419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 -- 172.21.15.145:0/2249884848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb419f610 con 0x7f0eb4104ad0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eb91da640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e9c0 tx=0x7f0ea400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400cd80 con 0x7f0eb4104ad0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea4005500 con 0x7f0eb4104ad0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb419f900 con 0x7f0eb4104ad0 2024-03-28T18:36:14.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4010690 con 0x7f0eb4104ad0 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.285+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb419fe50 con 0x7f0eb4104ad0 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.286+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb40fd620 con 0x7f0eb4104ad0 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.287+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0ea40046c0 con 0x7f0eb4104ad0 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.287+0000 7f0eaaffd640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 0x7f0ea00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.288+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0ea40cf1c0 con 0x7f0eb4104ad0 2024-03-28T18:36:14.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.288+0000 7f0eb99db640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 0x7f0ea00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.289 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.288+0000 7f0eb99db640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 0x7f0ea00a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0eb002c7b0 tx=0x7f0eb0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:14.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.292+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ea409bec0 con 0x7f0eb4104ad0 2024-03-28T18:36:14.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:14 smithi192 ceph-mon[36589]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:14 smithi192 ceph-mon[36589]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:14 smithi192 ceph-mon[36589]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T18:36:14.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:14 smithi192 ceph-mon[36589]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:14 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3218795608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:36:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:14 smithi164 ceph-mon[36248]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:14 smithi164 ceph-mon[36248]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:14 smithi164 ceph-mon[36248]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 2.2 KiB/s wr, 6 op/s 2024-03-28T18:36:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:14 smithi164 ceph-mon[36248]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:14.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:14 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3218795608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:36:14.470 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.470+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0eb4061ae0 con 0x7f0ea00a2890 2024-03-28T18:36:14.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.480+0000 7f0eaaffd640 1 -- 172.21.15.145:0/2249884848 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340501 (secure 0 0 0) 0x7f0eb4061ae0 con 0x7f0ea00a2890 2024-03-28T18:36:14.484 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:14.486 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-28T18:36:14.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 msgr2=0x7f0ea00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 0x7f0ea00a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0eb002c7b0 tx=0x7f0eb0005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 msgr2=0x7f0eb4197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e9c0 tx=0x7f0ea400ee80 comp rx=0 tx=0).stop 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 shutdown_connections 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eb4107400 0x7f0eb419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0eb4104ad0 0x7f0eb4197f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0ea00a2890 0x7f0ea00a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 --2- 172.21.15.145:0/2249884848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0eb41021a0 0x7f0eb41979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 >> 172.21.15.145:0/2249884848 conn(0x7f0eb40fbb30 msgr2=0x7f0eb4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 shutdown_connections 2024-03-28T18:36:14.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.484+0000 7f0ebbc3f640 1 -- 172.21.15.145:0/2249884848 wait complete. 2024-03-28T18:36:14.495 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-28T18:36:13.479332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596106,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":135,"num_read_kb":118,"num_write":299,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369404,"kb_used_data":11904,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572420,"statfs":{"total":1151940427776,"available":1151562158080,"internally_reserved":0,"allocated":12189696,"data_stored":7633362,"data_compressed":282522,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3328,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":73,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430944+0000","last_change":"2024-03-28T18:35:50.713920+0000","last_active":"2024-03-28T18:35:57.430944+0000","last_peered":"2024-03-28T18:35:57.430944+0000","last_clean":"2024-03-28T18:35:57.430944+0000","last_became_active":"2024-03-28T18:35:50.712723+0000","last_became_peered":"2024-03-28T18:35:50.712723+0000","last_unstale":"2024-03-28T18:35:57.430944+0000","last_undegraded":"2024-03-28T18:35:57.430944+0000","last_fullsized":"2024-03-28T18:35:57.430944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:31.562072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430655+0000","last_change":"2024-03-28T18:35:50.714271+0000","last_active":"2024-03-28T18:35:57.430655+0000","last_peered":"2024-03-28T18:35:57.430655+0000","last_clean":"2024-03-28T18:35:57.430655+0000","last_became_active":"2024-03-28T18:35:50.713461+0000","last_became_peered":"2024-03-28T18:35:50.713461+0000","last_unstale":"2024-03-28T18:35:57.430655+0000","last_undegraded":"2024-03-28T18:35:57.430655+0000","last_fullsized":"2024-03-28T18:35:57.430655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:03:47.789033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430680+0000","last_change":"2024-03-28T18:35:50.720511+0000","last_active":"2024-03-28T18:35:57.430680+0000","last_peered":"2024-03-28T18:35:57.430680+0000","last_clean":"2024-03-28T18:35:57.430680+0000","last_became_active":"2024-03-28T18:35:50.720227+0000","last_became_peered":"2024-03-28T18:35:50.720227+0000","last_unstale":"2024-03-28T18:35:57.430680+0000","last_undegraded":"2024-03-28T18:35:57.430680+0000","last_fullsized":"2024-03-28T18:35:57.430680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:19.849187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024086+0000","last_change":"2024-03-28T18:35:50.707852+0000","last_active":"2024-03-28T18:35:56.024086+0000","last_peered":"2024-03-28T18:35:56.024086+0000","last_clean":"2024-03-28T18:35:56.024086+0000","last_became_active":"2024-03-28T18:35:50.705465+0000","last_became_peered":"2024-03-28T18:35:50.705465+0000","last_unstale":"2024-03-28T18:35:56.024086+0000","last_undegraded":"2024-03-28T18:35:56.024086+0000","last_fullsized":"2024-03-28T18:35:56.024086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:00.251387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030447+0000","last_change":"2024-03-28T18:35:50.716142+0000","last_active":"2024-03-28T18:35:57.030447+0000","last_peered":"2024-03-28T18:35:57.030447+0000","last_clean":"2024-03-28T18:35:57.030447+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:35:57.030447+0000","last_undegraded":"2024-03-28T18:35:57.030447+0000","last_fullsized":"2024-03-28T18:35:57.030447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:46.515184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.793982+0000","last_change":"2024-03-28T18:35:50.712305+0000","last_active":"2024-03-28T18:35:56.793982+0000","last_peered":"2024-03-28T18:35:56.793982+0000","last_clean":"2024-03-28T18:35:56.793982+0000","last_became_active":"2024-03-28T18:35:50.711999+0000","last_became_peered":"2024-03-28T18:35:50.711999+0000","last_unstale":"2024-03-28T18:35:56.793982+0000","last_undegraded":"2024-03-28T18:35:56.793982+0000","last_fullsized":"2024-03-28T18:35:56.793982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:25:34.825605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430789+0000","last_change":"2024-03-28T18:35:50.713781+0000","last_active":"2024-03-28T18:35:57.430789+0000","last_peered":"2024-03-28T18:35:57.430789+0000","last_clean":"2024-03-28T18:35:57.430789+0000","last_became_active":"2024-03-28T18:35:50.713400+0000","last_became_peered":"2024-03-28T18:35:50.713400+0000","last_unstale":"2024-03-28T18:35:57.430789+0000","last_undegraded":"2024-03-28T18:35:57.430789+0000","last_fullsized":"2024-03-28T18:35:57.430789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:37:21.368701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379706+0000","last_change":"2024-03-28T18:35:50.704912+0000","last_active":"2024-03-28T18:35:56.379706+0000","last_peered":"2024-03-28T18:35:56.379706+0000","last_clean":"2024-03-28T18:35:56.379706+0000","last_became_active":"2024-03-28T18:35:50.704590+0000","last_became_peered":"2024-03-28T18:35:50.704590+0000","last_unstale":"2024-03-28T18:35:56.379706+0000","last_undegraded":"2024-03-28T18:35:56.379706+0000","last_fullsized":"2024-03-28T18:35:56.379706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:17.222377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431452+0000","last_change":"2024-03-28T18:35:50.710448+0000","last_active":"2024-03-28T18:35:57.431452+0000","last_peered":"2024-03-28T18:35:57.431452+0000","last_clean":"2024-03-28T18:35:57.431452+0000","last_became_active":"2024-03-28T18:35:50.708630+0000","last_became_peered":"2024-03-28T18:35:50.708630+0000","last_unstale":"2024-03-28T18:35:57.431452+0000","last_undegraded":"2024-03-28T18:35:57.431452+0000","last_fullsized":"2024-03-28T18:35:57.431452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:01.711677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794452+0000","last_change":"2024-03-28T18:35:50.711851+0000","last_active":"2024-03-28T18:35:56.794452+0000","last_peered":"2024-03-28T18:35:56.794452+0000","last_clean":"2024-03-28T18:35:56.794452+0000","last_became_active":"2024-03-28T18:35:50.711497+0000","last_became_peered":"2024-03-28T18:35:50.711497+0000","last_unstale":"2024-03-28T18:35:56.794452+0000","last_undegraded":"2024-03-28T18:35:56.794452+0000","last_fullsized":"2024-03-28T18:35:56.794452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:32.032471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431942+0000","last_change":"2024-03-28T18:35:50.722741+0000","last_active":"2024-03-28T18:35:57.431942+0000","last_peered":"2024-03-28T18:35:57.431942+0000","last_clean":"2024-03-28T18:35:57.431942+0000","last_became_active":"2024-03-28T18:35:50.722367+0000","last_became_peered":"2024-03-28T18:35:50.722367+0000","last_unstale":"2024-03-28T18:35:57.431942+0000","last_undegraded":"2024-03-28T18:35:57.431942+0000","last_fullsized":"2024-03-28T18:35:57.431942+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:28.898030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024082+0000","last_change":"2024-03-28T18:35:50.706167+0000","last_active":"2024-03-28T18:35:56.024082+0000","last_peered":"2024-03-28T18:35:56.024082+0000","last_clean":"2024-03-28T18:35:56.024082+0000","last_became_active":"2024-03-28T18:35:50.705485+0000","last_became_peered":"2024-03-28T18:35:50.705485+0000","last_unstale":"2024-03-28T18:35:56.024082+0000","last_undegraded":"2024-03-28T18:35:56.024082+0000","last_fullsized":"2024-03-28T18:35:56.024082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:57.492095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179564+0000","last_change":"2024-03-28T18:35:50.716920+0000","last_active":"2024-03-28T18:35:56.179564+0000","last_peered":"2024-03-28T18:35:56.179564+0000","last_clean":"2024-03-28T18:35:56.179564+0000","last_became_active":"2024-03-28T18:35:50.712942+0000","last_became_peered":"2024-03-28T18:35:50.712942+0000","last_unstale":"2024-03-28T18:35:56.179564+0000","last_undegraded":"2024-03-28T18:35:56.179564+0000","last_fullsized":"2024-03-28T18:35:56.179564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:22.550943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430404+0000","last_change":"2024-03-28T18:35:50.715257+0000","last_active":"2024-03-28T18:35:57.430404+0000","last_peered":"2024-03-28T18:35:57.430404+0000","last_clean":"2024-03-28T18:35:57.430404+0000","last_became_active":"2024-03-28T18:35:50.714943+0000","last_became_peered":"2024-03-28T18:35:50.714943+0000","last_unstale":"2024-03-28T18:35:57.430404+0000","last_undegraded":"2024-03-28T18:35:57.430404+0000","last_fullsized":"2024-03-28T18:35:57.430404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:39:05.683273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430583+0000","last_change":"2024-03-28T18:35:50.718042+0000","last_active":"2024-03-28T18:35:57.430583+0000","last_peered":"2024-03-28T18:35:57.430583+0000","last_clean":"2024-03-28T18:35:57.430583+0000","last_became_active":"2024-03-28T18:35:50.717713+0000","last_became_peered":"2024-03-28T18:35:50.717713+0000","last_unstale":"2024-03-28T18:35:57.430583+0000","last_undegraded":"2024-03-28T18:35:57.430583+0000","last_fullsized":"2024-03-28T18:35:57.430583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:51.764882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430663+0000","last_change":"2024-03-28T18:35:50.720741+0000","last_active":"2024-03-28T18:35:57.430663+0000","last_peered":"2024-03-28T18:35:57.430663+0000","last_clean":"2024-03-28T18:35:57.430663+0000","last_became_active":"2024-03-28T18:35:50.720038+0000","last_became_peered":"2024-03-28T18:35:50.720038+0000","last_unstale":"2024-03-28T18:35:57.430663+0000","last_undegraded":"2024-03-28T18:35:57.430663+0000","last_fullsized":"2024-03-28T18:35:57.430663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:48.001768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430846+0000","last_change":"2024-03-28T18:35:50.714152+0000","last_active":"2024-03-28T18:35:57.430846+0000","last_peered":"2024-03-28T18:35:57.430846+0000","last_clean":"2024-03-28T18:35:57.430846+0000","last_became_active":"2024-03-28T18:35:50.713813+0000","last_became_peered":"2024-03-28T18:35:50.713813+0000","last_unstale":"2024-03-28T18:35:57.430846+0000","last_undegraded":"2024-03-28T18:35:57.430846+0000","last_fullsized":"2024-03-28T18:35:57.430846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:48.698925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022952+0000","last_change":"2024-03-28T18:35:50.708046+0000","last_active":"2024-03-28T18:35:56.022952+0000","last_peered":"2024-03-28T18:35:56.022952+0000","last_clean":"2024-03-28T18:35:56.022952+0000","last_became_active":"2024-03-28T18:35:50.707694+0000","last_became_peered":"2024-03-28T18:35:50.707694+0000","last_unstale":"2024-03-28T18:35:56.022952+0000","last_undegraded":"2024-03-28T18:35:56.022952+0000","last_fullsized":"2024-03-28T18:35:56.022952+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:37:46.494020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023070+0000","last_change":"2024-03-28T18:35:50.714366+0000","last_active":"2024-03-28T18:35:56.023070+0000","last_peered":"2024-03-28T18:35:56.023070+0000","last_clean":"2024-03-28T18:35:56.023070+0000","last_became_active":"2024-03-28T18:35:50.714046+0000","last_became_peered":"2024-03-28T18:35:50.714046+0000","last_unstale":"2024-03-28T18:35:56.023070+0000","last_undegraded":"2024-03-28T18:35:56.023070+0000","last_fullsized":"2024-03-28T18:35:56.023070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:24.757304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022722+0000","last_change":"2024-03-28T18:35:55.032307+0000","last_active":"2024-03-28T18:35:56.022722+0000","last_peered":"2024-03-28T18:35:56.022722+0000","last_clean":"2024-03-28T18:35:56.022722+0000","last_became_active":"2024-03-28T18:35:55.031787+0000","last_became_peered":"2024-03-28T18:35:55.031787+0000","last_unstale":"2024-03-28T18:35:56.022722+0000","last_undegraded":"2024-03-28T18:35:56.022722+0000","last_fullsized":"2024-03-28T18:35:56.022722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:15.649053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023007+0000","last_change":"2024-03-28T18:35:49.933942+0000","last_active":"2024-03-28T18:35:56.023007+0000","last_peered":"2024-03-28T18:35:56.023007+0000","last_clean":"2024-03-28T18:35:56.023007+0000","last_became_active":"2024-03-28T18:35:49.933608+0000","last_became_peered":"2024-03-28T18:35:49.933608+0000","last_unstale":"2024-03-28T18:35:56.023007+0000","last_undegraded":"2024-03-28T18:35:56.023007+0000","last_fullsized":"2024-03-28T18:35:56.023007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:24.327806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380048+0000","last_change":"2024-03-28T18:35:50.704543+0000","last_active":"2024-03-28T18:35:56.380048+0000","last_peered":"2024-03-28T18:35:56.380048+0000","last_clean":"2024-03-28T18:35:56.380048+0000","last_became_active":"2024-03-28T18:35:50.704232+0000","last_became_peered":"2024-03-28T18:35:50.704232+0000","last_unstale":"2024-03-28T18:35:56.380048+0000","last_undegraded":"2024-03-28T18:35:56.380048+0000","last_fullsized":"2024-03-28T18:35:56.380048+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:30.934620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023573+0000","last_change":"2024-03-28T18:35:55.032545+0000","last_active":"2024-03-28T18:35:56.023573+0000","last_peered":"2024-03-28T18:35:56.023573+0000","last_clean":"2024-03-28T18:35:56.023573+0000","last_became_active":"2024-03-28T18:35:55.032011+0000","last_became_peered":"2024-03-28T18:35:55.032011+0000","last_unstale":"2024-03-28T18:35:56.023573+0000","last_undegraded":"2024-03-28T18:35:56.023573+0000","last_fullsized":"2024-03-28T18:35:56.023573+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:46.960816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380254+0000","last_change":"2024-03-28T18:35:49.940627+0000","last_active":"2024-03-28T18:35:56.380254+0000","last_peered":"2024-03-28T18:35:56.380254+0000","last_clean":"2024-03-28T18:35:56.380254+0000","last_became_active":"2024-03-28T18:35:49.940313+0000","last_became_peered":"2024-03-28T18:35:49.940313+0000","last_unstale":"2024-03-28T18:35:56.380254+0000","last_undegraded":"2024-03-28T18:35:56.380254+0000","last_fullsized":"2024-03-28T18:35:56.380254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:24.215713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430682+0000","last_change":"2024-03-28T18:35:50.720053+0000","last_active":"2024-03-28T18:35:57.430682+0000","last_peered":"2024-03-28T18:35:57.430682+0000","last_clean":"2024-03-28T18:35:57.430682+0000","last_became_active":"2024-03-28T18:35:50.719698+0000","last_became_peered":"2024-03-28T18:35:50.719698+0000","last_unstale":"2024-03-28T18:35:57.430682+0000","last_undegraded":"2024-03-28T18:35:57.430682+0000","last_fullsized":"2024-03-28T18:35:57.430682+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:30.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023864+0000","last_change":"2024-03-28T18:35:55.036468+0000","last_active":"2024-03-28T18:35:56.023864+0000","last_peered":"2024-03-28T18:35:56.023864+0000","last_clean":"2024-03-28T18:35:56.023864+0000","last_became_active":"2024-03-28T18:35:55.036104+0000","last_became_peered":"2024-03-28T18:35:55.036104+0000","last_unstale":"2024-03-28T18:35:56.023864+0000","last_undegraded":"2024-03-28T18:35:56.023864+0000","last_fullsized":"2024-03-28T18:35:56.023864+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:11:41.886642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.018309+0000","last_change":"2024-03-28T18:35:49.938447+0000","last_active":"2024-03-28T18:36:12.018309+0000","last_peered":"2024-03-28T18:36:12.018309+0000","last_clean":"2024-03-28T18:36:12.018309+0000","last_became_active":"2024-03-28T18:35:49.938125+0000","last_became_peered":"2024-03-28T18:35:49.938125+0000","last_unstale":"2024-03-28T18:36:12.018309+0000","last_undegraded":"2024-03-28T18:36:12.018309+0000","last_fullsized":"2024-03-28T18:36:12.018309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:31.064360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431381+0000","last_change":"2024-03-28T18:35:50.711208+0000","last_active":"2024-03-28T18:35:57.431381+0000","last_peered":"2024-03-28T18:35:57.431381+0000","last_clean":"2024-03-28T18:35:57.431381+0000","last_became_active":"2024-03-28T18:35:50.710306+0000","last_became_peered":"2024-03-28T18:35:50.710306+0000","last_unstale":"2024-03-28T18:35:57.431381+0000","last_undegraded":"2024-03-28T18:35:57.431381+0000","last_fullsized":"2024-03-28T18:35:57.431381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.449619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023605+0000","last_change":"2024-03-28T18:35:55.027968+0000","last_active":"2024-03-28T18:35:56.023605+0000","last_peered":"2024-03-28T18:35:56.023605+0000","last_clean":"2024-03-28T18:35:56.023605+0000","last_became_active":"2024-03-28T18:35:55.027486+0000","last_became_peered":"2024-03-28T18:35:55.027486+0000","last_unstale":"2024-03-28T18:35:56.023605+0000","last_undegraded":"2024-03-28T18:35:56.023605+0000","last_fullsized":"2024-03-28T18:35:56.023605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:30.088482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.017960+0000","last_change":"2024-03-28T18:35:49.936488+0000","last_active":"2024-03-28T18:36:12.017960+0000","last_peered":"2024-03-28T18:36:12.017960+0000","last_clean":"2024-03-28T18:36:12.017960+0000","last_became_active":"2024-03-28T18:35:49.936122+0000","last_became_peered":"2024-03-28T18:35:49.936122+0000","last_unstale":"2024-03-28T18:36:12.017960+0000","last_undegraded":"2024-03-28T18:36:12.017960+0000","last_fullsized":"2024-03-28T18:36:12.017960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:51.096079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431258+0000","last_change":"2024-03-28T18:35:50.704075+0000","last_active":"2024-03-28T18:35:57.431258+0000","last_peered":"2024-03-28T18:35:57.431258+0000","last_clean":"2024-03-28T18:35:57.431258+0000","last_became_active":"2024-03-28T18:35:50.703726+0000","last_became_peered":"2024-03-28T18:35:50.703726+0000","last_unstale":"2024-03-28T18:35:57.431258+0000","last_undegraded":"2024-03-28T18:35:57.431258+0000","last_fullsized":"2024-03-28T18:35:57.431258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:08.522098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023012+0000","last_change":"2024-03-28T18:35:55.023382+0000","last_active":"2024-03-28T18:35:56.023012+0000","last_peered":"2024-03-28T18:35:56.023012+0000","last_clean":"2024-03-28T18:35:56.023012+0000","last_became_active":"2024-03-28T18:35:55.022992+0000","last_became_peered":"2024-03-28T18:35:55.022992+0000","last_unstale":"2024-03-28T18:35:56.023012+0000","last_undegraded":"2024-03-28T18:35:56.023012+0000","last_fullsized":"2024-03-28T18:35:56.023012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:09.260173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430774+0000","last_change":"2024-03-28T18:35:49.939304+0000","last_active":"2024-03-28T18:35:57.430774+0000","last_peered":"2024-03-28T18:35:57.430774+0000","last_clean":"2024-03-28T18:35:57.430774+0000","last_became_active":"2024-03-28T18:35:49.938976+0000","last_became_peered":"2024-03-28T18:35:49.938976+0000","last_unstale":"2024-03-28T18:35:57.430774+0000","last_undegraded":"2024-03-28T18:35:57.430774+0000","last_fullsized":"2024-03-28T18:35:57.430774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:37.368257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030593+0000","last_change":"2024-03-28T18:35:50.715841+0000","last_active":"2024-03-28T18:35:57.030593+0000","last_peered":"2024-03-28T18:35:57.030593+0000","last_clean":"2024-03-28T18:35:57.030593+0000","last_became_active":"2024-03-28T18:35:50.715365+0000","last_became_peered":"2024-03-28T18:35:50.715365+0000","last_unstale":"2024-03-28T18:35:57.030593+0000","last_undegraded":"2024-03-28T18:35:57.030593+0000","last_fullsized":"2024-03-28T18:35:57.030593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:33.009378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023001+0000","last_change":"2024-03-28T18:35:55.033073+0000","last_active":"2024-03-28T18:35:56.023001+0000","last_peered":"2024-03-28T18:35:56.023001+0000","last_clean":"2024-03-28T18:35:56.023001+0000","last_became_active":"2024-03-28T18:35:55.032649+0000","last_became_peered":"2024-03-28T18:35:55.032649+0000","last_unstale":"2024-03-28T18:35:56.023001+0000","last_undegraded":"2024-03-28T18:35:56.023001+0000","last_fullsized":"2024-03-28T18:35:56.023001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:14.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:10.091208+0000","last_change":"2024-03-28T18:35:49.932672+0000","last_active":"2024-03-28T18:36:10.091208+0000","last_peered":"2024-03-28T18:36:10.091208+0000","last_clean":"2024-03-28T18:36:10.091208+0000","last_became_active":"2024-03-28T18:35:49.932207+0000","last_became_peered":"2024-03-28T18:35:49.932207+0000","last_unstale":"2024-03-28T18:36:10.091208+0000","last_undegraded":"2024-03-28T18:36:10.091208+0000","last_fullsized":"2024-03-28T18:36:10.091208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:50.606075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430902+0000","last_change":"2024-03-28T18:35:50.696535+0000","last_active":"2024-03-28T18:35:57.430902+0000","last_peered":"2024-03-28T18:35:57.430902+0000","last_clean":"2024-03-28T18:35:57.430902+0000","last_became_active":"2024-03-28T18:35:50.696142+0000","last_became_peered":"2024-03-28T18:35:50.696142+0000","last_unstale":"2024-03-28T18:35:57.430902+0000","last_undegraded":"2024-03-28T18:35:57.430902+0000","last_fullsized":"2024-03-28T18:35:57.430902+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:24:59.749302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430866+0000","last_change":"2024-03-28T18:35:55.032203+0000","last_active":"2024-03-28T18:35:57.430866+0000","last_peered":"2024-03-28T18:35:57.430866+0000","last_clean":"2024-03-28T18:35:57.430866+0000","last_became_active":"2024-03-28T18:35:55.031734+0000","last_became_peered":"2024-03-28T18:35:55.031734+0000","last_unstale":"2024-03-28T18:35:57.430866+0000","last_undegraded":"2024-03-28T18:35:57.430866+0000","last_fullsized":"2024-03-28T18:35:57.430866+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:08.224595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:10.092750+0000","last_change":"2024-03-28T18:35:49.929798+0000","last_active":"2024-03-28T18:36:10.092750+0000","last_peered":"2024-03-28T18:36:10.092750+0000","last_clean":"2024-03-28T18:36:10.092750+0000","last_became_active":"2024-03-28T18:35:49.929278+0000","last_became_peered":"2024-03-28T18:35:49.929278+0000","last_unstale":"2024-03-28T18:36:10.092750+0000","last_undegraded":"2024-03-28T18:36:10.092750+0000","last_fullsized":"2024-03-28T18:36:10.092750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:33:13.078225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379574+0000","last_change":"2024-03-28T18:35:50.702935+0000","last_active":"2024-03-28T18:35:56.379574+0000","last_peered":"2024-03-28T18:35:56.379574+0000","last_clean":"2024-03-28T18:35:56.379574+0000","last_became_active":"2024-03-28T18:35:50.702586+0000","last_became_peered":"2024-03-28T18:35:50.702586+0000","last_unstale":"2024-03-28T18:35:56.379574+0000","last_undegraded":"2024-03-28T18:35:56.379574+0000","last_fullsized":"2024-03-28T18:35:56.379574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:50.335605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430722+0000","last_change":"2024-03-28T18:35:55.028541+0000","last_active":"2024-03-28T18:35:57.430722+0000","last_peered":"2024-03-28T18:35:57.430722+0000","last_clean":"2024-03-28T18:35:57.430722+0000","last_became_active":"2024-03-28T18:35:55.028142+0000","last_became_peered":"2024-03-28T18:35:55.028142+0000","last_unstale":"2024-03-28T18:35:57.430722+0000","last_undegraded":"2024-03-28T18:35:57.430722+0000","last_fullsized":"2024-03-28T18:35:57.430722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:20.738071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430486+0000","last_change":"2024-03-28T18:35:49.938255+0000","last_active":"2024-03-28T18:35:57.430486+0000","last_peered":"2024-03-28T18:35:57.430486+0000","last_clean":"2024-03-28T18:35:57.430486+0000","last_became_active":"2024-03-28T18:35:49.937930+0000","last_became_peered":"2024-03-28T18:35:49.937930+0000","last_unstale":"2024-03-28T18:35:57.430486+0000","last_undegraded":"2024-03-28T18:35:57.430486+0000","last_fullsized":"2024-03-28T18:35:57.430486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:41.621941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022472+0000","last_change":"2024-03-28T18:35:50.706437+0000","last_active":"2024-03-28T18:35:56.022472+0000","last_peered":"2024-03-28T18:35:56.022472+0000","last_clean":"2024-03-28T18:35:56.022472+0000","last_became_active":"2024-03-28T18:35:50.706139+0000","last_became_peered":"2024-03-28T18:35:50.706139+0000","last_unstale":"2024-03-28T18:35:56.022472+0000","last_undegraded":"2024-03-28T18:35:56.022472+0000","last_fullsized":"2024-03-28T18:35:56.022472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:52.750896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379906+0000","last_change":"2024-03-28T18:35:55.025289+0000","last_active":"2024-03-28T18:35:56.379906+0000","last_peered":"2024-03-28T18:35:56.379906+0000","last_clean":"2024-03-28T18:35:56.379906+0000","last_became_active":"2024-03-28T18:35:55.024835+0000","last_became_peered":"2024-03-28T18:35:55.024835+0000","last_unstale":"2024-03-28T18:35:56.379906+0000","last_undegraded":"2024-03-28T18:35:56.379906+0000","last_fullsized":"2024-03-28T18:35:56.379906+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:08.933559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.080816+0000","last_change":"2024-03-28T18:35:49.940518+0000","last_active":"2024-03-28T18:36:09.080816+0000","last_peered":"2024-03-28T18:36:09.080816+0000","last_clean":"2024-03-28T18:36:09.080816+0000","last_became_active":"2024-03-28T18:35:49.940208+0000","last_became_peered":"2024-03-28T18:35:49.940208+0000","last_unstale":"2024-03-28T18:36:09.080816+0000","last_undegraded":"2024-03-28T18:36:09.080816+0000","last_fullsized":"2024-03-28T18:36:09.080816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:55.557840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431590+0000","last_change":"2024-03-28T18:35:50.710781+0000","last_active":"2024-03-28T18:35:57.431590+0000","last_peered":"2024-03-28T18:35:57.431590+0000","last_clean":"2024-03-28T18:35:57.431590+0000","last_became_active":"2024-03-28T18:35:50.710458+0000","last_became_peered":"2024-03-28T18:35:50.710458+0000","last_unstale":"2024-03-28T18:35:57.431590+0000","last_undegraded":"2024-03-28T18:35:57.431590+0000","last_fullsized":"2024-03-28T18:35:57.431590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:49.020331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022622+0000","last_change":"2024-03-28T18:35:55.034023+0000","last_active":"2024-03-28T18:35:56.022622+0000","last_peered":"2024-03-28T18:35:56.022622+0000","last_clean":"2024-03-28T18:35:56.022622+0000","last_became_active":"2024-03-28T18:35:55.033634+0000","last_became_peered":"2024-03-28T18:35:55.033634+0000","last_unstale":"2024-03-28T18:35:56.022622+0000","last_undegraded":"2024-03-28T18:35:56.022622+0000","last_fullsized":"2024-03-28T18:35:56.022622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:41.221061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431709+0000","last_change":"2024-03-28T18:35:49.937443+0000","last_active":"2024-03-28T18:35:57.431709+0000","last_peered":"2024-03-28T18:35:57.431709+0000","last_clean":"2024-03-28T18:35:57.431709+0000","last_became_active":"2024-03-28T18:35:49.937003+0000","last_became_peered":"2024-03-28T18:35:49.937003+0000","last_unstale":"2024-03-28T18:35:57.431709+0000","last_undegraded":"2024-03-28T18:35:57.431709+0000","last_fullsized":"2024-03-28T18:35:57.431709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:49:05.804868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430721+0000","last_change":"2024-03-28T18:35:50.718480+0000","last_active":"2024-03-28T18:35:57.430721+0000","last_peered":"2024-03-28T18:35:57.430721+0000","last_clean":"2024-03-28T18:35:57.430721+0000","last_became_active":"2024-03-28T18:35:50.717253+0000","last_became_peered":"2024-03-28T18:35:50.717253+0000","last_unstale":"2024-03-28T18:35:57.430721+0000","last_undegraded":"2024-03-28T18:35:57.430721+0000","last_fullsized":"2024-03-28T18:35:57.430721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:59:30.071806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024149+0000","last_change":"2024-03-28T18:35:55.032461+0000","last_active":"2024-03-28T18:35:56.024149+0000","last_peered":"2024-03-28T18:35:56.024149+0000","last_clean":"2024-03-28T18:35:56.024149+0000","last_became_active":"2024-03-28T18:35:55.032029+0000","last_became_peered":"2024-03-28T18:35:55.032029+0000","last_unstale":"2024-03-28T18:35:56.024149+0000","last_undegraded":"2024-03-28T18:35:56.024149+0000","last_fullsized":"2024-03-28T18:35:56.024149+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:27:11.216371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431223+0000","last_change":"2024-03-28T18:35:49.934357+0000","last_active":"2024-03-28T18:35:57.431223+0000","last_peered":"2024-03-28T18:35:57.431223+0000","last_clean":"2024-03-28T18:35:57.431223+0000","last_became_active":"2024-03-28T18:35:49.933994+0000","last_became_peered":"2024-03-28T18:35:49.933994+0000","last_unstale":"2024-03-28T18:35:57.431223+0000","last_undegraded":"2024-03-28T18:35:57.431223+0000","last_fullsized":"2024-03-28T18:35:57.431223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:21.408585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431532+0000","last_change":"2024-03-28T18:35:50.716771+0000","last_active":"2024-03-28T18:35:57.431532+0000","last_peered":"2024-03-28T18:35:57.431532+0000","last_clean":"2024-03-28T18:35:57.431532+0000","last_became_active":"2024-03-28T18:35:50.716393+0000","last_became_peered":"2024-03-28T18:35:50.716393+0000","last_unstale":"2024-03-28T18:35:57.431532+0000","last_undegraded":"2024-03-28T18:35:57.431532+0000","last_fullsized":"2024-03-28T18:35:57.431532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:33:10.881475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022479+0000","last_change":"2024-03-28T18:35:55.027650+0000","last_active":"2024-03-28T18:35:56.022479+0000","last_peered":"2024-03-28T18:35:56.022479+0000","last_clean":"2024-03-28T18:35:56.022479+0000","last_became_active":"2024-03-28T18:35:55.027109+0000","last_became_peered":"2024-03-28T18:35:55.027109+0000","last_unstale":"2024-03-28T18:35:56.022479+0000","last_undegraded":"2024-03-28T18:35:56.022479+0000","last_fullsized":"2024-03-28T18:35:56.022479+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:54.826415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.083443+0000","last_change":"2024-03-28T18:35:49.934499+0000","last_active":"2024-03-28T18:36:09.083443+0000","last_peered":"2024-03-28T18:36:09.083443+0000","last_clean":"2024-03-28T18:36:09.083443+0000","last_became_active":"2024-03-28T18:35:49.934150+0000","last_became_peered":"2024-03-28T18:35:49.934150+0000","last_unstale":"2024-03-28T18:36:09.083443+0000","last_undegraded":"2024-03-28T18:36:09.083443+0000","last_fullsized":"2024-03-28T18:36:09.083443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:53.661593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030694+0000","last_change":"2024-03-28T18:35:50.714162+0000","last_active":"2024-03-28T18:35:57.030694+0000","last_peered":"2024-03-28T18:35:57.030694+0000","last_clean":"2024-03-28T18:35:57.030694+0000","last_became_active":"2024-03-28T18:35:50.713819+0000","last_became_peered":"2024-03-28T18:35:50.713819+0000","last_unstale":"2024-03-28T18:35:57.030694+0000","last_undegraded":"2024-03-28T18:35:57.030694+0000","last_fullsized":"2024-03-28T18:35:57.030694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:49:19.365559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431591+0000","last_change":"2024-03-28T18:35:55.028460+0000","last_active":"2024-03-28T18:35:57.431591+0000","last_peered":"2024-03-28T18:35:57.431591+0000","last_clean":"2024-03-28T18:35:57.431591+0000","last_became_active":"2024-03-28T18:35:55.028015+0000","last_became_peered":"2024-03-28T18:35:55.028015+0000","last_unstale":"2024-03-28T18:35:57.431591+0000","last_undegraded":"2024-03-28T18:35:57.431591+0000","last_fullsized":"2024-03-28T18:35:57.431591+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:00.864611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022804+0000","last_change":"2024-03-28T18:35:49.938090+0000","last_active":"2024-03-28T18:35:56.022804+0000","last_peered":"2024-03-28T18:35:56.022804+0000","last_clean":"2024-03-28T18:35:56.022804+0000","last_became_active":"2024-03-28T18:35:49.937572+0000","last_became_peered":"2024-03-28T18:35:49.937572+0000","last_unstale":"2024-03-28T18:35:56.022804+0000","last_undegraded":"2024-03-28T18:35:56.022804+0000","last_fullsized":"2024-03-28T18:35:56.022804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:41.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179442+0000","last_change":"2024-03-28T18:35:50.717286+0000","last_active":"2024-03-28T18:35:56.179442+0000","last_peered":"2024-03-28T18:35:56.179442+0000","last_clean":"2024-03-28T18:35:56.179442+0000","last_became_active":"2024-03-28T18:35:50.712917+0000","last_became_peered":"2024-03-28T18:35:50.712917+0000","last_unstale":"2024-03-28T18:35:56.179442+0000","last_undegraded":"2024-03-28T18:35:56.179442+0000","last_fullsized":"2024-03-28T18:35:56.179442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:10:42.306028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430732+0000","last_change":"2024-03-28T18:35:55.023384+0000","last_active":"2024-03-28T18:35:57.430732+0000","last_peered":"2024-03-28T18:35:57.430732+0000","last_clean":"2024-03-28T18:35:57.430732+0000","last_became_active":"2024-03-28T18:35:55.023013+0000","last_became_peered":"2024-03-28T18:35:55.023013+0000","last_unstale":"2024-03-28T18:35:57.430732+0000","last_undegraded":"2024-03-28T18:35:57.430732+0000","last_fullsized":"2024-03-28T18:35:57.430732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:01.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431150+0000","last_change":"2024-03-28T18:35:49.939073+0000","last_active":"2024-03-28T18:35:57.431150+0000","last_peered":"2024-03-28T18:35:57.431150+0000","last_clean":"2024-03-28T18:35:57.431150+0000","last_became_active":"2024-03-28T18:35:49.938764+0000","last_became_peered":"2024-03-28T18:35:49.938764+0000","last_unstale":"2024-03-28T18:35:57.431150+0000","last_undegraded":"2024-03-28T18:35:57.431150+0000","last_fullsized":"2024-03-28T18:35:57.431150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:52.089059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430987+0000","last_change":"2024-03-28T18:35:50.716132+0000","last_active":"2024-03-28T18:35:57.430987+0000","last_peered":"2024-03-28T18:35:57.430987+0000","last_clean":"2024-03-28T18:35:57.430987+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:35:57.430987+0000","last_undegraded":"2024-03-28T18:35:57.430987+0000","last_fullsized":"2024-03-28T18:35:57.430987+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:57.930739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022880+0000","last_change":"2024-03-28T18:35:55.035553+0000","last_active":"2024-03-28T18:35:56.022880+0000","last_peered":"2024-03-28T18:35:56.022880+0000","last_clean":"2024-03-28T18:35:56.022880+0000","last_became_active":"2024-03-28T18:35:55.035134+0000","last_became_peered":"2024-03-28T18:35:55.035134+0000","last_unstale":"2024-03-28T18:35:56.022880+0000","last_undegraded":"2024-03-28T18:35:56.022880+0000","last_fullsized":"2024-03-28T18:35:56.022880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:03:52.976361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023453+0000","last_change":"2024-03-28T18:35:49.932696+0000","last_active":"2024-03-28T18:35:56.023453+0000","last_peered":"2024-03-28T18:35:56.023453+0000","last_clean":"2024-03-28T18:35:56.023453+0000","last_became_active":"2024-03-28T18:35:49.932350+0000","last_became_peered":"2024-03-28T18:35:49.932350+0000","last_unstale":"2024-03-28T18:35:56.023453+0000","last_undegraded":"2024-03-28T18:35:56.023453+0000","last_fullsized":"2024-03-28T18:35:56.023453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:26.183549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430791+0000","last_change":"2024-03-28T18:35:50.712734+0000","last_active":"2024-03-28T18:35:57.430791+0000","last_peered":"2024-03-28T18:35:57.430791+0000","last_clean":"2024-03-28T18:35:57.430791+0000","last_became_active":"2024-03-28T18:35:50.712409+0000","last_became_peered":"2024-03-28T18:35:50.712409+0000","last_unstale":"2024-03-28T18:35:57.430791+0000","last_undegraded":"2024-03-28T18:35:57.430791+0000","last_fullsized":"2024-03-28T18:35:57.430791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:36.344182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179901+0000","last_change":"2024-03-28T18:35:55.025047+0000","last_active":"2024-03-28T18:35:56.179901+0000","last_peered":"2024-03-28T18:35:56.179901+0000","last_clean":"2024-03-28T18:35:56.179901+0000","last_became_active":"2024-03-28T18:35:55.024652+0000","last_became_peered":"2024-03-28T18:35:55.024652+0000","last_unstale":"2024-03-28T18:35:56.179901+0000","last_undegraded":"2024-03-28T18:35:56.179901+0000","last_fullsized":"2024-03-28T18:35:56.179901+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:17.750951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.008930+0000","last_change":"2024-03-28T18:35:49.940506+0000","last_active":"2024-03-28T18:36:12.008930+0000","last_peered":"2024-03-28T18:36:12.008930+0000","last_clean":"2024-03-28T18:36:12.008930+0000","last_became_active":"2024-03-28T18:35:49.940168+0000","last_became_peered":"2024-03-28T18:35:49.940168+0000","last_unstale":"2024-03-28T18:36:12.008930+0000","last_undegraded":"2024-03-28T18:36:12.008930+0000","last_fullsized":"2024-03-28T18:36:12.008930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:47.629255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430825+0000","last_change":"2024-03-28T18:35:50.705023+0000","last_active":"2024-03-28T18:35:57.430825+0000","last_peered":"2024-03-28T18:35:57.430825+0000","last_clean":"2024-03-28T18:35:57.430825+0000","last_became_active":"2024-03-28T18:35:50.704694+0000","last_became_peered":"2024-03-28T18:35:50.704694+0000","last_unstale":"2024-03-28T18:35:57.430825+0000","last_undegraded":"2024-03-28T18:35:57.430825+0000","last_fullsized":"2024-03-28T18:35:57.430825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:46.677189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430770+0000","last_change":"2024-03-28T18:35:55.028311+0000","last_active":"2024-03-28T18:35:57.430770+0000","last_peered":"2024-03-28T18:35:57.430770+0000","last_clean":"2024-03-28T18:35:57.430770+0000","last_became_active":"2024-03-28T18:35:55.027914+0000","last_became_peered":"2024-03-28T18:35:55.027914+0000","last_unstale":"2024-03-28T18:35:57.430770+0000","last_undegraded":"2024-03-28T18:35:57.430770+0000","last_fullsized":"2024-03-28T18:35:57.430770+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:59.684230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:11.086917+0000","last_change":"2024-03-28T18:35:49.938004+0000","last_active":"2024-03-28T18:36:11.086917+0000","last_peered":"2024-03-28T18:36:11.086917+0000","last_clean":"2024-03-28T18:36:11.086917+0000","last_became_active":"2024-03-28T18:35:49.937615+0000","last_became_peered":"2024-03-28T18:35:49.937615+0000","last_unstale":"2024-03-28T18:36:11.086917+0000","last_undegraded":"2024-03-28T18:36:11.086917+0000","last_fullsized":"2024-03-28T18:36:11.086917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:21.619857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430623+0000","last_change":"2024-03-28T18:35:50.719276+0000","last_active":"2024-03-28T18:35:57.430623+0000","last_peered":"2024-03-28T18:35:57.430623+0000","last_clean":"2024-03-28T18:35:57.430623+0000","last_became_active":"2024-03-28T18:35:50.718896+0000","last_became_peered":"2024-03-28T18:35:50.718896+0000","last_unstale":"2024-03-28T18:35:57.430623+0000","last_undegraded":"2024-03-28T18:35:57.430623+0000","last_fullsized":"2024-03-28T18:35:57.430623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:24.494713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030708+0000","last_change":"2024-03-28T18:35:55.099193+0000","last_active":"2024-03-28T18:35:57.030708+0000","last_peered":"2024-03-28T18:35:57.030708+0000","last_clean":"2024-03-28T18:35:57.030708+0000","last_became_active":"2024-03-28T18:35:55.098795+0000","last_became_peered":"2024-03-28T18:35:55.098795+0000","last_unstale":"2024-03-28T18:35:57.030708+0000","last_undegraded":"2024-03-28T18:35:57.030708+0000","last_fullsized":"2024-03-28T18:35:57.030708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:07.872554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022664+0000","last_change":"2024-03-28T18:35:49.936885+0000","last_active":"2024-03-28T18:35:56.022664+0000","last_peered":"2024-03-28T18:35:56.022664+0000","last_clean":"2024-03-28T18:35:56.022664+0000","last_became_active":"2024-03-28T18:35:49.936563+0000","last_became_peered":"2024-03-28T18:35:49.936563+0000","last_unstale":"2024-03-28T18:35:56.022664+0000","last_undegraded":"2024-03-28T18:35:56.022664+0000","last_fullsized":"2024-03-28T18:35:56.022664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:33.631160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794247+0000","last_change":"2024-03-28T18:35:50.713750+0000","last_active":"2024-03-28T18:35:56.794247+0000","last_peered":"2024-03-28T18:35:56.794247+0000","last_clean":"2024-03-28T18:35:56.794247+0000","last_became_active":"2024-03-28T18:35:50.713378+0000","last_became_peered":"2024-03-28T18:35:50.713378+0000","last_unstale":"2024-03-28T18:35:56.794247+0000","last_undegraded":"2024-03-28T18:35:56.794247+0000","last_fullsized":"2024-03-28T18:35:56.794247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:36.072651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179597+0000","last_change":"2024-03-28T18:35:55.030827+0000","last_active":"2024-03-28T18:35:56.179597+0000","last_peered":"2024-03-28T18:35:56.179597+0000","last_clean":"2024-03-28T18:35:56.179597+0000","last_became_active":"2024-03-28T18:35:55.030382+0000","last_became_peered":"2024-03-28T18:35:55.030382+0000","last_unstale":"2024-03-28T18:35:56.179597+0000","last_undegraded":"2024-03-28T18:35:56.179597+0000","last_fullsized":"2024-03-28T18:35:56.179597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:41:58.371568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431032+0000","last_change":"2024-03-28T18:35:49.932290+0000","last_active":"2024-03-28T18:35:57.431032+0000","last_peered":"2024-03-28T18:35:57.431032+0000","last_clean":"2024-03-28T18:35:57.431032+0000","last_became_active":"2024-03-28T18:35:49.931925+0000","last_became_peered":"2024-03-28T18:35:49.931925+0000","last_unstale":"2024-03-28T18:35:57.431032+0000","last_undegraded":"2024-03-28T18:35:57.431032+0000","last_fullsized":"2024-03-28T18:35:57.431032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:26.477117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794044+0000","last_change":"2024-03-28T18:35:50.713654+0000","last_active":"2024-03-28T18:35:56.794044+0000","last_peered":"2024-03-28T18:35:56.794044+0000","last_clean":"2024-03-28T18:35:56.794044+0000","last_became_active":"2024-03-28T18:35:50.713146+0000","last_became_peered":"2024-03-28T18:35:50.713146+0000","last_unstale":"2024-03-28T18:35:56.794044+0000","last_undegraded":"2024-03-28T18:35:56.794044+0000","last_fullsized":"2024-03-28T18:35:56.794044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:23.951963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179820+0000","last_change":"2024-03-28T18:35:55.099331+0000","last_active":"2024-03-28T18:35:56.179820+0000","last_peered":"2024-03-28T18:35:56.179820+0000","last_clean":"2024-03-28T18:35:56.179820+0000","last_became_active":"2024-03-28T18:35:55.098912+0000","last_became_peered":"2024-03-28T18:35:55.098912+0000","last_unstale":"2024-03-28T18:35:56.179820+0000","last_undegraded":"2024-03-28T18:35:56.179820+0000","last_fullsized":"2024-03-28T18:35:56.179820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:31.082148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430700+0000","last_change":"2024-03-28T18:35:50.704377+0000","last_active":"2024-03-28T18:35:57.430700+0000","last_peered":"2024-03-28T18:35:57.430700+0000","last_clean":"2024-03-28T18:35:57.430700+0000","last_became_active":"2024-03-28T18:35:50.703991+0000","last_became_peered":"2024-03-28T18:35:50.703991+0000","last_unstale":"2024-03-28T18:35:57.430700+0000","last_undegraded":"2024-03-28T18:35:57.430700+0000","last_fullsized":"2024-03-28T18:35:57.430700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:48.173466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015411+0000","last_change":"2024-03-28T18:35:49.933528+0000","last_active":"2024-03-28T18:36:12.015411+0000","last_peered":"2024-03-28T18:36:12.015411+0000","last_clean":"2024-03-28T18:36:12.015411+0000","last_became_active":"2024-03-28T18:35:49.933171+0000","last_became_peered":"2024-03-28T18:35:49.933171+0000","last_unstale":"2024-03-28T18:36:12.015411+0000","last_undegraded":"2024-03-28T18:36:12.015411+0000","last_fullsized":"2024-03-28T18:36:12.015411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:50:59.928400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431778+0000","last_change":"2024-03-28T18:35:55.029752+0000","last_active":"2024-03-28T18:35:57.431778+0000","last_peered":"2024-03-28T18:35:57.431778+0000","last_clean":"2024-03-28T18:35:57.431778+0000","last_became_active":"2024-03-28T18:35:55.029407+0000","last_became_peered":"2024-03-28T18:35:55.029407+0000","last_unstale":"2024-03-28T18:35:57.431778+0000","last_undegraded":"2024-03-28T18:35:57.431778+0000","last_fullsized":"2024-03-28T18:35:57.431778+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:17.328315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379713+0000","last_change":"2024-03-28T18:35:50.706380+0000","last_active":"2024-03-28T18:35:56.379713+0000","last_peered":"2024-03-28T18:35:56.379713+0000","last_clean":"2024-03-28T18:35:56.379713+0000","last_became_active":"2024-03-28T18:35:50.706044+0000","last_became_peered":"2024-03-28T18:35:50.706044+0000","last_unstale":"2024-03-28T18:35:56.379713+0000","last_undegraded":"2024-03-28T18:35:56.379713+0000","last_fullsized":"2024-03-28T18:35:56.379713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:04.916646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023655+0000","last_change":"2024-03-28T18:35:49.930241+0000","last_active":"2024-03-28T18:35:56.023655+0000","last_peered":"2024-03-28T18:35:56.023655+0000","last_clean":"2024-03-28T18:35:56.023655+0000","last_became_active":"2024-03-28T18:35:49.929830+0000","last_became_peered":"2024-03-28T18:35:49.929830+0000","last_unstale":"2024-03-28T18:35:56.023655+0000","last_undegraded":"2024-03-28T18:35:56.023655+0000","last_fullsized":"2024-03-28T18:35:56.023655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:11.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431044+0000","last_change":"2024-03-28T18:34:38.804796+0000","last_active":"2024-03-28T18:35:57.431044+0000","last_peered":"2024-03-28T18:35:57.431044+0000","last_clean":"2024-03-28T18:35:57.431044+0000","last_became_active":"2024-03-28T18:34:38.804419+0000","last_became_peered":"2024-03-28T18:34:38.804419+0000","last_unstale":"2024-03-28T18:35:57.431044+0000","last_undegraded":"2024-03-28T18:35:57.431044+0000","last_fullsized":"2024-03-28T18:35:57.431044+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_clean_scrub_stamp":"2024-03-28T18:31:13.733610+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:38.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431487+0000","last_change":"2024-03-28T18:35:55.099390+0000","last_active":"2024-03-28T18:35:57.431487+0000","last_peered":"2024-03-28T18:35:57.431487+0000","last_clean":"2024-03-28T18:35:57.431487+0000","last_became_active":"2024-03-28T18:35:55.098960+0000","last_became_peered":"2024-03-28T18:35:55.098960+0000","last_unstale":"2024-03-28T18:35:57.431487+0000","last_undegraded":"2024-03-28T18:35:57.431487+0000","last_fullsized":"2024-03-28T18:35:57.431487+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:22.460846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179226+0000","last_change":"2024-03-28T18:35:50.711961+0000","last_active":"2024-03-28T18:35:56.179226+0000","last_peered":"2024-03-28T18:35:56.179226+0000","last_clean":"2024-03-28T18:35:56.179226+0000","last_became_active":"2024-03-28T18:35:50.705466+0000","last_became_peered":"2024-03-28T18:35:50.705466+0000","last_unstale":"2024-03-28T18:35:56.179226+0000","last_undegraded":"2024-03-28T18:35:56.179226+0000","last_fullsized":"2024-03-28T18:35:56.179226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:02.527220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430673+0000","last_change":"2024-03-28T18:35:49.936938+0000","last_active":"2024-03-28T18:35:57.430673+0000","last_peered":"2024-03-28T18:35:57.430673+0000","last_clean":"2024-03-28T18:35:57.430673+0000","last_became_active":"2024-03-28T18:35:49.936562+0000","last_became_peered":"2024-03-28T18:35:49.936562+0000","last_unstale":"2024-03-28T18:35:57.430673+0000","last_undegraded":"2024-03-28T18:35:57.430673+0000","last_fullsized":"2024-03-28T18:35:57.430673+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:18.067381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430893+0000","last_change":"2024-03-28T18:35:55.023402+0000","last_active":"2024-03-28T18:35:57.430893+0000","last_peered":"2024-03-28T18:35:57.430893+0000","last_clean":"2024-03-28T18:35:57.430893+0000","last_became_active":"2024-03-28T18:35:55.023006+0000","last_became_peered":"2024-03-28T18:35:55.023006+0000","last_unstale":"2024-03-28T18:35:57.430893+0000","last_undegraded":"2024-03-28T18:35:57.430893+0000","last_fullsized":"2024-03-28T18:35:57.430893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:12.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.008832+0000","last_change":"2024-03-28T18:35:49.942494+0000","last_active":"2024-03-28T18:36:12.008832+0000","last_peered":"2024-03-28T18:36:12.008832+0000","last_clean":"2024-03-28T18:36:12.008832+0000","last_became_active":"2024-03-28T18:35:49.941607+0000","last_became_peered":"2024-03-28T18:35:49.941607+0000","last_unstale":"2024-03-28T18:36:12.008832+0000","last_undegraded":"2024-03-28T18:36:12.008832+0000","last_fullsized":"2024-03-28T18:36:12.008832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:25.315100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179646+0000","last_change":"2024-03-28T18:35:50.707912+0000","last_active":"2024-03-28T18:35:56.179646+0000","last_peered":"2024-03-28T18:35:56.179646+0000","last_clean":"2024-03-28T18:35:56.179646+0000","last_became_active":"2024-03-28T18:35:50.707304+0000","last_became_peered":"2024-03-28T18:35:50.707304+0000","last_unstale":"2024-03-28T18:35:56.179646+0000","last_undegraded":"2024-03-28T18:35:56.179646+0000","last_fullsized":"2024-03-28T18:35:56.179646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:45.556370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379711+0000","last_change":"2024-03-28T18:35:55.099919+0000","last_active":"2024-03-28T18:35:56.379711+0000","last_peered":"2024-03-28T18:35:56.379711+0000","last_clean":"2024-03-28T18:35:56.379711+0000","last_became_active":"2024-03-28T18:35:55.099430+0000","last_became_peered":"2024-03-28T18:35:55.099430+0000","last_unstale":"2024-03-28T18:35:56.379711+0000","last_undegraded":"2024-03-28T18:35:56.379711+0000","last_fullsized":"2024-03-28T18:35:56.379711+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:30.091473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022950+0000","last_change":"2024-03-28T18:35:49.935871+0000","last_active":"2024-03-28T18:35:56.022950+0000","last_peered":"2024-03-28T18:35:56.022950+0000","last_clean":"2024-03-28T18:35:56.022950+0000","last_became_active":"2024-03-28T18:35:49.935536+0000","last_became_peered":"2024-03-28T18:35:49.935536+0000","last_unstale":"2024-03-28T18:35:56.022950+0000","last_undegraded":"2024-03-28T18:35:56.022950+0000","last_fullsized":"2024-03-28T18:35:56.022950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:22.915067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380161+0000","last_change":"2024-03-28T18:35:50.710877+0000","last_active":"2024-03-28T18:35:56.380161+0000","last_peered":"2024-03-28T18:35:56.380161+0000","last_clean":"2024-03-28T18:35:56.380161+0000","last_became_active":"2024-03-28T18:35:50.709987+0000","last_became_peered":"2024-03-28T18:35:50.709987+0000","last_unstale":"2024-03-28T18:35:56.380161+0000","last_undegraded":"2024-03-28T18:35:56.380161+0000","last_fullsized":"2024-03-28T18:35:56.380161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:43.092541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022648+0000","last_change":"2024-03-28T18:35:55.023390+0000","last_active":"2024-03-28T18:35:56.022648+0000","last_peered":"2024-03-28T18:35:56.022648+0000","last_clean":"2024-03-28T18:35:56.022648+0000","last_became_active":"2024-03-28T18:35:55.023034+0000","last_became_peered":"2024-03-28T18:35:55.023034+0000","last_unstale":"2024-03-28T18:35:56.022648+0000","last_undegraded":"2024-03-28T18:35:56.022648+0000","last_fullsized":"2024-03-28T18:35:56.022648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:05.829144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024232+0000","last_change":"2024-03-28T18:35:49.932675+0000","last_active":"2024-03-28T18:35:56.024232+0000","last_peered":"2024-03-28T18:35:56.024232+0000","last_clean":"2024-03-28T18:35:56.024232+0000","last_became_active":"2024-03-28T18:35:49.932246+0000","last_became_peered":"2024-03-28T18:35:49.932246+0000","last_unstale":"2024-03-28T18:35:56.024232+0000","last_undegraded":"2024-03-28T18:35:56.024232+0000","last_fullsized":"2024-03-28T18:35:56.024232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:46.806411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430754+0000","last_change":"2024-03-28T18:35:50.716861+0000","last_active":"2024-03-28T18:35:57.430754+0000","last_peered":"2024-03-28T18:35:57.430754+0000","last_clean":"2024-03-28T18:35:57.430754+0000","last_became_active":"2024-03-28T18:35:50.716377+0000","last_became_peered":"2024-03-28T18:35:50.716377+0000","last_unstale":"2024-03-28T18:35:57.430754+0000","last_undegraded":"2024-03-28T18:35:57.430754+0000","last_fullsized":"2024-03-28T18:35:57.430754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:27.018809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023989+0000","last_change":"2024-03-28T18:35:55.099891+0000","last_active":"2024-03-28T18:35:56.023989+0000","last_peered":"2024-03-28T18:35:56.023989+0000","last_clean":"2024-03-28T18:35:56.023989+0000","last_became_active":"2024-03-28T18:35:55.098988+0000","last_became_peered":"2024-03-28T18:35:55.098988+0000","last_unstale":"2024-03-28T18:35:56.023989+0000","last_undegraded":"2024-03-28T18:35:56.023989+0000","last_fullsized":"2024-03-28T18:35:56.023989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:15.682731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.082560+0000","last_change":"2024-03-28T18:35:49.936146+0000","last_active":"2024-03-28T18:36:09.082560+0000","last_peered":"2024-03-28T18:36:09.082560+0000","last_clean":"2024-03-28T18:36:09.082560+0000","last_became_active":"2024-03-28T18:35:49.935771+0000","last_became_peered":"2024-03-28T18:35:49.935771+0000","last_unstale":"2024-03-28T18:36:09.082560+0000","last_undegraded":"2024-03-28T18:36:09.082560+0000","last_fullsized":"2024-03-28T18:36:09.082560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:58.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030376+0000","last_change":"2024-03-28T18:35:50.707527+0000","last_active":"2024-03-28T18:35:57.030376+0000","last_peered":"2024-03-28T18:35:57.030376+0000","last_clean":"2024-03-28T18:35:57.030376+0000","last_became_active":"2024-03-28T18:35:50.707216+0000","last_became_peered":"2024-03-28T18:35:50.707216+0000","last_unstale":"2024-03-28T18:35:57.030376+0000","last_undegraded":"2024-03-28T18:35:57.030376+0000","last_fullsized":"2024-03-28T18:35:57.030376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:32.344898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022955+0000","last_change":"2024-03-28T18:35:55.033294+0000","last_active":"2024-03-28T18:35:56.022955+0000","last_peered":"2024-03-28T18:35:56.022955+0000","last_clean":"2024-03-28T18:35:56.022955+0000","last_became_active":"2024-03-28T18:35:55.030743+0000","last_became_peered":"2024-03-28T18:35:55.030743+0000","last_unstale":"2024-03-28T18:35:56.022955+0000","last_undegraded":"2024-03-28T18:35:56.022955+0000","last_fullsized":"2024-03-28T18:35:56.022955+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:16:11.003542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015863+0000","last_change":"2024-03-28T18:35:49.931394+0000","last_active":"2024-03-28T18:36:12.015863+0000","last_peered":"2024-03-28T18:36:12.015863+0000","last_clean":"2024-03-28T18:36:12.015863+0000","last_became_active":"2024-03-28T18:35:49.931045+0000","last_became_peered":"2024-03-28T18:35:49.931045+0000","last_unstale":"2024-03-28T18:36:12.015863+0000","last_undegraded":"2024-03-28T18:36:12.015863+0000","last_fullsized":"2024-03-28T18:36:12.015863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:54.030481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430381+0000","last_change":"2024-03-28T18:35:50.714745+0000","last_active":"2024-03-28T18:35:57.430381+0000","last_peered":"2024-03-28T18:35:57.430381+0000","last_clean":"2024-03-28T18:35:57.430381+0000","last_became_active":"2024-03-28T18:35:50.712991+0000","last_became_peered":"2024-03-28T18:35:50.712991+0000","last_unstale":"2024-03-28T18:35:57.430381+0000","last_undegraded":"2024-03-28T18:35:57.430381+0000","last_fullsized":"2024-03-28T18:35:57.430381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:34.403662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023963+0000","last_change":"2024-03-28T18:35:55.032353+0000","last_active":"2024-03-28T18:35:56.023963+0000","last_peered":"2024-03-28T18:35:56.023963+0000","last_clean":"2024-03-28T18:35:56.023963+0000","last_became_active":"2024-03-28T18:35:55.031889+0000","last_became_peered":"2024-03-28T18:35:55.031889+0000","last_unstale":"2024-03-28T18:35:56.023963+0000","last_undegraded":"2024-03-28T18:35:56.023963+0000","last_fullsized":"2024-03-28T18:35:56.023963+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:56.358941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022597+0000","last_change":"2024-03-28T18:35:49.944107+0000","last_active":"2024-03-28T18:35:56.022597+0000","last_peered":"2024-03-28T18:35:56.022597+0000","last_clean":"2024-03-28T18:35:56.022597+0000","last_became_active":"2024-03-28T18:35:49.943726+0000","last_became_peered":"2024-03-28T18:35:49.943726+0000","last_unstale":"2024-03-28T18:35:56.022597+0000","last_undegraded":"2024-03-28T18:35:56.022597+0000","last_fullsized":"2024-03-28T18:35:56.022597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:05.301302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379995+0000","last_change":"2024-03-28T18:35:50.709236+0000","last_active":"2024-03-28T18:35:56.379995+0000","last_peered":"2024-03-28T18:35:56.379995+0000","last_clean":"2024-03-28T18:35:56.379995+0000","last_became_active":"2024-03-28T18:35:50.708824+0000","last_became_peered":"2024-03-28T18:35:50.708824+0000","last_unstale":"2024-03-28T18:35:56.379995+0000","last_undegraded":"2024-03-28T18:35:56.379995+0000","last_fullsized":"2024-03-28T18:35:56.379995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:21:30.207033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024213+0000","last_change":"2024-03-28T18:35:55.099711+0000","last_active":"2024-03-28T18:35:56.024213+0000","last_peered":"2024-03-28T18:35:56.024213+0000","last_clean":"2024-03-28T18:35:56.024213+0000","last_became_active":"2024-03-28T18:35:55.099054+0000","last_became_peered":"2024-03-28T18:35:55.099054+0000","last_unstale":"2024-03-28T18:35:56.024213+0000","last_undegraded":"2024-03-28T18:35:56.024213+0000","last_fullsized":"2024-03-28T18:35:56.024213+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:22:32.326655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.082594+0000","last_change":"2024-03-28T18:35:49.939428+0000","last_active":"2024-03-28T18:36:09.082594+0000","last_peered":"2024-03-28T18:36:09.082594+0000","last_clean":"2024-03-28T18:36:09.082594+0000","last_became_active":"2024-03-28T18:35:49.939089+0000","last_became_peered":"2024-03-28T18:35:49.939089+0000","last_unstale":"2024-03-28T18:36:09.082594+0000","last_undegraded":"2024-03-28T18:36:09.082594+0000","last_fullsized":"2024-03-28T18:36:09.082594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:38.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431212+0000","last_change":"2024-03-28T18:35:50.720931+0000","last_active":"2024-03-28T18:35:57.431212+0000","last_peered":"2024-03-28T18:35:57.431212+0000","last_clean":"2024-03-28T18:35:57.431212+0000","last_became_active":"2024-03-28T18:35:50.720544+0000","last_became_peered":"2024-03-28T18:35:50.720544+0000","last_unstale":"2024-03-28T18:35:57.431212+0000","last_undegraded":"2024-03-28T18:35:57.431212+0000","last_fullsized":"2024-03-28T18:35:57.431212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:05.092688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431122+0000","last_change":"2024-03-28T18:35:55.024518+0000","last_active":"2024-03-28T18:35:57.431122+0000","last_peered":"2024-03-28T18:35:57.431122+0000","last_clean":"2024-03-28T18:35:57.431122+0000","last_became_active":"2024-03-28T18:35:55.024103+0000","last_became_peered":"2024-03-28T18:35:55.024103+0000","last_unstale":"2024-03-28T18:35:57.431122+0000","last_undegraded":"2024-03-28T18:35:57.431122+0000","last_fullsized":"2024-03-28T18:35:57.431122+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:32:57.620744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023097+0000","last_change":"2024-03-28T18:35:49.943828+0000","last_active":"2024-03-28T18:35:56.023097+0000","last_peered":"2024-03-28T18:35:56.023097+0000","last_clean":"2024-03-28T18:35:56.023097+0000","last_became_active":"2024-03-28T18:35:49.941729+0000","last_became_peered":"2024-03-28T18:35:49.941729+0000","last_unstale":"2024-03-28T18:35:56.023097+0000","last_undegraded":"2024-03-28T18:35:56.023097+0000","last_fullsized":"2024-03-28T18:35:56.023097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:22.989447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431953+0000","last_change":"2024-03-28T18:35:50.720449+0000","last_active":"2024-03-28T18:35:57.431953+0000","last_peered":"2024-03-28T18:35:57.431953+0000","last_clean":"2024-03-28T18:35:57.431953+0000","last_became_active":"2024-03-28T18:35:50.719840+0000","last_became_peered":"2024-03-28T18:35:50.719840+0000","last_unstale":"2024-03-28T18:35:57.431953+0000","last_undegraded":"2024-03-28T18:35:57.431953+0000","last_fullsized":"2024-03-28T18:35:57.431953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:48.980677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430486+0000","last_change":"2024-03-28T18:35:55.032614+0000","last_active":"2024-03-28T18:35:57.430486+0000","last_peered":"2024-03-28T18:35:57.430486+0000","last_clean":"2024-03-28T18:35:57.430486+0000","last_became_active":"2024-03-28T18:35:55.032180+0000","last_became_peered":"2024-03-28T18:35:55.032180+0000","last_unstale":"2024-03-28T18:35:57.430486+0000","last_undegraded":"2024-03-28T18:35:57.430486+0000","last_fullsized":"2024-03-28T18:35:57.430486+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:40.121016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431318+0000","last_change":"2024-03-28T18:35:49.940681+0000","last_active":"2024-03-28T18:35:57.431318+0000","last_peered":"2024-03-28T18:35:57.431318+0000","last_clean":"2024-03-28T18:35:57.431318+0000","last_became_active":"2024-03-28T18:35:49.940376+0000","last_became_peered":"2024-03-28T18:35:49.940376+0000","last_unstale":"2024-03-28T18:35:57.431318+0000","last_undegraded":"2024-03-28T18:35:57.431318+0000","last_fullsized":"2024-03-28T18:35:57.431318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:02:11.926285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431414+0000","last_change":"2024-03-28T18:35:50.715051+0000","last_active":"2024-03-28T18:35:57.431414+0000","last_peered":"2024-03-28T18:35:57.431414+0000","last_clean":"2024-03-28T18:35:57.431414+0000","last_became_active":"2024-03-28T18:35:50.714647+0000","last_became_peered":"2024-03-28T18:35:50.714647+0000","last_unstale":"2024-03-28T18:35:57.431414+0000","last_undegraded":"2024-03-28T18:35:57.431414+0000","last_fullsized":"2024-03-28T18:35:57.431414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:02.832665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431313+0000","last_change":"2024-03-28T18:35:55.021869+0000","last_active":"2024-03-28T18:35:57.431313+0000","last_peered":"2024-03-28T18:35:57.431313+0000","last_clean":"2024-03-28T18:35:57.431313+0000","last_became_active":"2024-03-28T18:35:55.021535+0000","last_became_peered":"2024-03-28T18:35:55.021535+0000","last_unstale":"2024-03-28T18:35:57.431313+0000","last_undegraded":"2024-03-28T18:35:57.431313+0000","last_fullsized":"2024-03-28T18:35:57.431313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:31.610882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015038+0000","last_change":"2024-03-28T18:35:49.936095+0000","last_active":"2024-03-28T18:36:12.015038+0000","last_peered":"2024-03-28T18:36:12.015038+0000","last_clean":"2024-03-28T18:36:12.015038+0000","last_became_active":"2024-03-28T18:35:49.935766+0000","last_became_peered":"2024-03-28T18:35:49.935766+0000","last_unstale":"2024-03-28T18:36:12.015038+0000","last_undegraded":"2024-03-28T18:36:12.015038+0000","last_fullsized":"2024-03-28T18:36:12.015038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:00:06.712979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431846+0000","last_change":"2024-03-28T18:35:50.715313+0000","last_active":"2024-03-28T18:35:57.431846+0000","last_peered":"2024-03-28T18:35:57.431846+0000","last_clean":"2024-03-28T18:35:57.431846+0000","last_became_active":"2024-03-28T18:35:50.714923+0000","last_became_peered":"2024-03-28T18:35:50.714923+0000","last_unstale":"2024-03-28T18:35:57.431846+0000","last_undegraded":"2024-03-28T18:35:57.431846+0000","last_fullsized":"2024-03-28T18:35:57.431846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:49.406551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430435+0000","last_change":"2024-03-28T18:35:50.714244+0000","last_active":"2024-03-28T18:35:57.430435+0000","last_peered":"2024-03-28T18:35:57.430435+0000","last_clean":"2024-03-28T18:35:57.430435+0000","last_became_active":"2024-03-28T18:35:50.713347+0000","last_became_peered":"2024-03-28T18:35:50.713347+0000","last_unstale":"2024-03-28T18:35:57.430435+0000","last_undegraded":"2024-03-28T18:35:57.430435+0000","last_fullsized":"2024-03-28T18:35:57.430435+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:34.360341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022920+0000","last_change":"2024-03-28T18:35:50.711132+0000","last_active":"2024-03-28T18:35:56.022920+0000","last_peered":"2024-03-28T18:35:56.022920+0000","last_clean":"2024-03-28T18:35:56.022920+0000","last_became_active":"2024-03-28T18:35:50.710500+0000","last_became_peered":"2024-03-28T18:35:50.710500+0000","last_unstale":"2024-03-28T18:35:56.022920+0000","last_undegraded":"2024-03-28T18:35:56.022920+0000","last_fullsized":"2024-03-28T18:35:56.022920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:22.060106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022766+0000","last_change":"2024-03-28T18:35:50.712396+0000","last_active":"2024-03-28T18:35:56.022766+0000","last_peered":"2024-03-28T18:35:56.022766+0000","last_clean":"2024-03-28T18:35:56.022766+0000","last_became_active":"2024-03-28T18:35:50.711950+0000","last_became_peered":"2024-03-28T18:35:50.711950+0000","last_unstale":"2024-03-28T18:35:56.022766+0000","last_undegraded":"2024-03-28T18:35:56.022766+0000","last_fullsized":"2024-03-28T18:35:56.022766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:24.573754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430688+0000","last_change":"2024-03-28T18:35:50.718222+0000","last_active":"2024-03-28T18:35:57.430688+0000","last_peered":"2024-03-28T18:35:57.430688+0000","last_clean":"2024-03-28T18:35:57.430688+0000","last_became_active":"2024-03-28T18:35:50.717234+0000","last_became_peered":"2024-03-28T18:35:50.717234+0000","last_unstale":"2024-03-28T18:35:57.430688+0000","last_undegraded":"2024-03-28T18:35:57.430688+0000","last_fullsized":"2024-03-28T18:35:57.430688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:28:47.544280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179593+0000","last_change":"2024-03-28T18:35:50.707486+0000","last_active":"2024-03-28T18:35:56.179593+0000","last_peered":"2024-03-28T18:35:56.179593+0000","last_clean":"2024-03-28T18:35:56.179593+0000","last_became_active":"2024-03-28T18:35:50.706902+0000","last_became_peered":"2024-03-28T18:35:50.706902+0000","last_unstale":"2024-03-28T18:35:56.179593+0000","last_undegraded":"2024-03-28T18:35:56.179593+0000","last_fullsized":"2024-03-28T18:35:56.179593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:00.160873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430769+0000","last_change":"2024-03-28T18:35:50.715566+0000","last_active":"2024-03-28T18:35:57.430769+0000","last_peered":"2024-03-28T18:35:57.430769+0000","last_clean":"2024-03-28T18:35:57.430769+0000","last_became_active":"2024-03-28T18:35:50.715193+0000","last_became_peered":"2024-03-28T18:35:50.715193+0000","last_unstale":"2024-03-28T18:35:57.430769+0000","last_undegraded":"2024-03-28T18:35:57.430769+0000","last_fullsized":"2024-03-28T18:35:57.430769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:57.126250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023032+0000","last_change":"2024-03-28T18:35:50.709904+0000","last_active":"2024-03-28T18:35:56.023032+0000","last_peered":"2024-03-28T18:35:56.023032+0000","last_clean":"2024-03-28T18:35:56.023032+0000","last_became_active":"2024-03-28T18:35:50.709575+0000","last_became_peered":"2024-03-28T18:35:50.709575+0000","last_unstale":"2024-03-28T18:35:56.023032+0000","last_undegraded":"2024-03-28T18:35:56.023032+0000","last_fullsized":"2024-03-28T18:35:56.023032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:26.580718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380260+0000","last_change":"2024-03-28T18:35:50.705804+0000","last_active":"2024-03-28T18:35:56.380260+0000","last_peered":"2024-03-28T18:35:56.380260+0000","last_clean":"2024-03-28T18:35:56.380260+0000","last_became_active":"2024-03-28T18:35:50.705482+0000","last_became_peered":"2024-03-28T18:35:50.705482+0000","last_unstale":"2024-03-28T18:35:56.380260+0000","last_undegraded":"2024-03-28T18:35:56.380260+0000","last_fullsized":"2024-03-28T18:35:56.380260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:37.512184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430586+0000","last_change":"2024-03-28T18:35:50.707563+0000","last_active":"2024-03-28T18:35:57.430586+0000","last_peered":"2024-03-28T18:35:57.430586+0000","last_clean":"2024-03-28T18:35:57.430586+0000","last_became_active":"2024-03-28T18:35:50.706300+0000","last_became_peered":"2024-03-28T18:35:50.706300+0000","last_unstale":"2024-03-28T18:35:57.430586+0000","last_undegraded":"2024-03-28T18:35:57.430586+0000","last_fullsized":"2024-03-28T18:35:57.430586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:05.324365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379875+0000","last_change":"2024-03-28T18:35:50.705549+0000","last_active":"2024-03-28T18:35:56.379875+0000","last_peered":"2024-03-28T18:35:56.379875+0000","last_clean":"2024-03-28T18:35:56.379875+0000","last_became_active":"2024-03-28T18:35:50.705190+0000","last_became_peered":"2024-03-28T18:35:50.705190+0000","last_unstale":"2024-03-28T18:35:56.379875+0000","last_undegraded":"2024-03-28T18:35:56.379875+0000","last_fullsized":"2024-03-28T18:35:56.379875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:38.662227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431512+0000","last_change":"2024-03-28T18:35:50.719591+0000","last_active":"2024-03-28T18:35:57.431512+0000","last_peered":"2024-03-28T18:35:57.431512+0000","last_clean":"2024-03-28T18:35:57.431512+0000","last_became_active":"2024-03-28T18:35:50.719272+0000","last_became_peered":"2024-03-28T18:35:50.719272+0000","last_unstale":"2024-03-28T18:35:57.431512+0000","last_undegraded":"2024-03-28T18:35:57.431512+0000","last_fullsized":"2024-03-28T18:35:57.431512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:06.627761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179444+0000","last_change":"2024-03-28T18:35:50.712965+0000","last_active":"2024-03-28T18:35:56.179444+0000","last_peered":"2024-03-28T18:35:56.179444+0000","last_clean":"2024-03-28T18:35:56.179444+0000","last_became_active":"2024-03-28T18:35:50.712648+0000","last_became_peered":"2024-03-28T18:35:50.712648+0000","last_unstale":"2024-03-28T18:35:56.179444+0000","last_undegraded":"2024-03-28T18:35:56.179444+0000","last_fullsized":"2024-03-28T18:35:56.179444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:00.925664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430626+0000","last_change":"2024-03-28T18:35:50.717277+0000","last_active":"2024-03-28T18:35:57.430626+0000","last_peered":"2024-03-28T18:35:57.430626+0000","last_clean":"2024-03-28T18:35:57.430626+0000","last_became_active":"2024-03-28T18:35:50.716161+0000","last_became_peered":"2024-03-28T18:35:50.716161+0000","last_unstale":"2024-03-28T18:35:57.430626+0000","last_undegraded":"2024-03-28T18:35:57.430626+0000","last_fullsized":"2024-03-28T18:35:57.430626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:42.789575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210208,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5738,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":118,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":19426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318288,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":487067,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579926,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":1082233,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487851,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005088,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":487805,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266725,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":1078017,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":487789,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":490909,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":488545,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":490943,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":487497,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":991232,"data_stored":487089,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724590,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":1077617,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:36:14.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:36:14.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 -- 172.21.15.145:0/2450443178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb928103290 msgr2=0x7fb928109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 --2- 172.21.15.145:0/2450443178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb928103290 0x7fb928109b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb920009ca0 tx=0x7fb92002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 -- 172.21.15.145:0/2450443178 shutdown_connections 2024-03-28T18:36:14.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 --2- 172.21.15.145:0/2450443178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb928103290 0x7fb928109b20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 --2- 172.21.15.145:0/2450443178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 --2- 172.21.15.145:0/2450443178 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9281016e0 0x7fb928101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 -- 172.21.15.145:0/2450443178 >> 172.21.15.145:0/2450443178 conn(0x7fb9280fcc50 msgr2=0x7fb9280ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 -- 172.21.15.145:0/2450443178 shutdown_connections 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.637+0000 7fb92f87e640 1 -- 172.21.15.145:0/2450443178 wait complete. 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 Processor -- start 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 -- start start 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 0x7fb928197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92810d360 con 0x7fb928103290 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92810d1e0 con 0x7fb9281016e0 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92f87e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92810d4e0 con 0x7fb9281028d0 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92de1b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92d61a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 0x7fb928197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92ce19640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92de1b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58132/0 (socket says 172.21.15.145:58132) 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.638+0000 7fb92de1b640 1 -- 172.21.15.145:0/1549737199 learned_addr learned my addr 172.21.15.145:0/1549737199 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 -- 172.21.15.145:0/1549737199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 msgr2=0x7fb928197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 -- 172.21.15.145:0/1549737199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 msgr2=0x7fb928197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 0x7fb928197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 -- 172.21.15.145:0/1549737199 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92819f630 con 0x7fb928103290 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92ce19640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92d61a640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 0x7fb928197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92de1b640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb920032530 tx=0x7fb920032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920041020 con 0x7fb928103290 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92819f8c0 con 0x7fb928103290 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb920034cd0 con 0x7fb928103290 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920039040 con 0x7fb928103290 2024-03-28T18:36:14.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.639+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92819fd80 con 0x7fb928103290 2024-03-28T18:36:14.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.640+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9281089a0 con 0x7fb928103290 2024-03-28T18:36:14.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.642+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb920044020 con 0x7fb928103290 2024-03-28T18:36:14.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.643+0000 7fb91e7fc640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 0x7fb9140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:14.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.643+0000 7fb92d61a640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 0x7fb9140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:14.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.643+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb9200f0b60 con 0x7fb928103290 2024-03-28T18:36:14.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.643+0000 7fb92d61a640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 0x7fb9140a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb928102730 tx=0x7fb924005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:14.647 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.647+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb92004b020 con 0x7fb928103290 2024-03-28T18:36:14.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.828+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb928061ae0 con 0x7fb9140a2890 2024-03-28T18:36:14.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.838+0000 7fb91e7fc640 1 -- 172.21.15.145:0/1549737199 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340501 (secure 0 0 0) 0x7fb928061ae0 con 0x7fb9140a2890 2024-03-28T18:36:14.841 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:14.843 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 msgr2=0x7fb9140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 0x7fb9140a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb928102730 tx=0x7fb924005f90 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 msgr2=0x7fb92819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb920032530 tx=0x7fb920032f20 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 shutdown_connections 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb928103290 0x7fb92819efc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9140a2890 0x7fb9140a4d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9281028d0 0x7fb928197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 --2- 172.21.15.145:0/1549737199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9281016e0 0x7fb928197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 >> 172.21.15.145:0/1549737199 conn(0x7fb9280fcc50 msgr2=0x7fb928105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 shutdown_connections 2024-03-28T18:36:14.845 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:14.842+0000 7fb92f87e640 1 -- 172.21.15.145:0/1549737199 wait complete. 2024-03-28T18:36:14.853 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-28T18:36:13.479332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596106,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":135,"num_read_kb":118,"num_write":299,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369404,"kb_used_data":11904,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572420,"statfs":{"total":1151940427776,"available":1151562158080,"internally_reserved":0,"allocated":12189696,"data_stored":7633362,"data_compressed":282522,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3328,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":73,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430944+0000","last_change":"2024-03-28T18:35:50.713920+0000","last_active":"2024-03-28T18:35:57.430944+0000","last_peered":"2024-03-28T18:35:57.430944+0000","last_clean":"2024-03-28T18:35:57.430944+0000","last_became_active":"2024-03-28T18:35:50.712723+0000","last_became_peered":"2024-03-28T18:35:50.712723+0000","last_unstale":"2024-03-28T18:35:57.430944+0000","last_undegraded":"2024-03-28T18:35:57.430944+0000","last_fullsized":"2024-03-28T18:35:57.430944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:31.562072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430655+0000","last_change":"2024-03-28T18:35:50.714271+0000","last_active":"2024-03-28T18:35:57.430655+0000","last_peered":"2024-03-28T18:35:57.430655+0000","last_clean":"2024-03-28T18:35:57.430655+0000","last_became_active":"2024-03-28T18:35:50.713461+0000","last_became_peered":"2024-03-28T18:35:50.713461+0000","last_unstale":"2024-03-28T18:35:57.430655+0000","last_undegraded":"2024-03-28T18:35:57.430655+0000","last_fullsized":"2024-03-28T18:35:57.430655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:03:47.789033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430680+0000","last_change":"2024-03-28T18:35:50.720511+0000","last_active":"2024-03-28T18:35:57.430680+0000","last_peered":"2024-03-28T18:35:57.430680+0000","last_clean":"2024-03-28T18:35:57.430680+0000","last_became_active":"2024-03-28T18:35:50.720227+0000","last_became_peered":"2024-03-28T18:35:50.720227+0000","last_unstale":"2024-03-28T18:35:57.430680+0000","last_undegraded":"2024-03-28T18:35:57.430680+0000","last_fullsized":"2024-03-28T18:35:57.430680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:19.849187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024086+0000","last_change":"2024-03-28T18:35:50.707852+0000","last_active":"2024-03-28T18:35:56.024086+0000","last_peered":"2024-03-28T18:35:56.024086+0000","last_clean":"2024-03-28T18:35:56.024086+0000","last_became_active":"2024-03-28T18:35:50.705465+0000","last_became_peered":"2024-03-28T18:35:50.705465+0000","last_unstale":"2024-03-28T18:35:56.024086+0000","last_undegraded":"2024-03-28T18:35:56.024086+0000","last_fullsized":"2024-03-28T18:35:56.024086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:00.251387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030447+0000","last_change":"2024-03-28T18:35:50.716142+0000","last_active":"2024-03-28T18:35:57.030447+0000","last_peered":"2024-03-28T18:35:57.030447+0000","last_clean":"2024-03-28T18:35:57.030447+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:35:57.030447+0000","last_undegraded":"2024-03-28T18:35:57.030447+0000","last_fullsized":"2024-03-28T18:35:57.030447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:46.515184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.793982+0000","last_change":"2024-03-28T18:35:50.712305+0000","last_active":"2024-03-28T18:35:56.793982+0000","last_peered":"2024-03-28T18:35:56.793982+0000","last_clean":"2024-03-28T18:35:56.793982+0000","last_became_active":"2024-03-28T18:35:50.711999+0000","last_became_peered":"2024-03-28T18:35:50.711999+0000","last_unstale":"2024-03-28T18:35:56.793982+0000","last_undegraded":"2024-03-28T18:35:56.793982+0000","last_fullsized":"2024-03-28T18:35:56.793982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:25:34.825605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430789+0000","last_change":"2024-03-28T18:35:50.713781+0000","last_active":"2024-03-28T18:35:57.430789+0000","last_peered":"2024-03-28T18:35:57.430789+0000","last_clean":"2024-03-28T18:35:57.430789+0000","last_became_active":"2024-03-28T18:35:50.713400+0000","last_became_peered":"2024-03-28T18:35:50.713400+0000","last_unstale":"2024-03-28T18:35:57.430789+0000","last_undegraded":"2024-03-28T18:35:57.430789+0000","last_fullsized":"2024-03-28T18:35:57.430789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:37:21.368701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379706+0000","last_change":"2024-03-28T18:35:50.704912+0000","last_active":"2024-03-28T18:35:56.379706+0000","last_peered":"2024-03-28T18:35:56.379706+0000","last_clean":"2024-03-28T18:35:56.379706+0000","last_became_active":"2024-03-28T18:35:50.704590+0000","last_became_peered":"2024-03-28T18:35:50.704590+0000","last_unstale":"2024-03-28T18:35:56.379706+0000","last_undegraded":"2024-03-28T18:35:56.379706+0000","last_fullsized":"2024-03-28T18:35:56.379706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:17.222377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431452+0000","last_change":"2024-03-28T18:35:50.710448+0000","last_active":"2024-03-28T18:35:57.431452+0000","last_peered":"2024-03-28T18:35:57.431452+0000","last_clean":"2024-03-28T18:35:57.431452+0000","last_became_active":"2024-03-28T18:35:50.708630+0000","last_became_peered":"2024-03-28T18:35:50.708630+0000","last_unstale":"2024-03-28T18:35:57.431452+0000","last_undegraded":"2024-03-28T18:35:57.431452+0000","last_fullsized":"2024-03-28T18:35:57.431452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:01.711677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794452+0000","last_change":"2024-03-28T18:35:50.711851+0000","last_active":"2024-03-28T18:35:56.794452+0000","last_peered":"2024-03-28T18:35:56.794452+0000","last_clean":"2024-03-28T18:35:56.794452+0000","last_became_active":"2024-03-28T18:35:50.711497+0000","last_became_peered":"2024-03-28T18:35:50.711497+0000","last_unstale":"2024-03-28T18:35:56.794452+0000","last_undegraded":"2024-03-28T18:35:56.794452+0000","last_fullsized":"2024-03-28T18:35:56.794452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:32.032471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431942+0000","last_change":"2024-03-28T18:35:50.722741+0000","last_active":"2024-03-28T18:35:57.431942+0000","last_peered":"2024-03-28T18:35:57.431942+0000","last_clean":"2024-03-28T18:35:57.431942+0000","last_became_active":"2024-03-28T18:35:50.722367+0000","last_became_peered":"2024-03-28T18:35:50.722367+0000","last_unstale":"2024-03-28T18:35:57.431942+0000","last_undegraded":"2024-03-28T18:35:57.431942+0000","last_fullsized":"2024-03-28T18:35:57.431942+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:28.898030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024082+0000","last_change":"2024-03-28T18:35:50.706167+0000","last_active":"2024-03-28T18:35:56.024082+0000","last_peered":"2024-03-28T18:35:56.024082+0000","last_clean":"2024-03-28T18:35:56.024082+0000","last_became_active":"2024-03-28T18:35:50.705485+0000","last_became_peered":"2024-03-28T18:35:50.705485+0000","last_unstale":"2024-03-28T18:35:56.024082+0000","last_undegraded":"2024-03-28T18:35:56.024082+0000","last_fullsized":"2024-03-28T18:35:56.024082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:57.492095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179564+0000","last_change":"2024-03-28T18:35:50.716920+0000","last_active":"2024-03-28T18:35:56.179564+0000","last_peered":"2024-03-28T18:35:56.179564+0000","last_clean":"2024-03-28T18:35:56.179564+0000","last_became_active":"2024-03-28T18:35:50.712942+0000","last_became_peered":"2024-03-28T18:35:50.712942+0000","last_unstale":"2024-03-28T18:35:56.179564+0000","last_undegraded":"2024-03-28T18:35:56.179564+0000","last_fullsized":"2024-03-28T18:35:56.179564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:22.550943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430404+0000","last_change":"2024-03-28T18:35:50.715257+0000","last_active":"2024-03-28T18:35:57.430404+0000","last_peered":"2024-03-28T18:35:57.430404+0000","last_clean":"2024-03-28T18:35:57.430404+0000","last_became_active":"2024-03-28T18:35:50.714943+0000","last_became_peered":"2024-03-28T18:35:50.714943+0000","last_unstale":"2024-03-28T18:35:57.430404+0000","last_undegraded":"2024-03-28T18:35:57.430404+0000","last_fullsized":"2024-03-28T18:35:57.430404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:39:05.683273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430583+0000","last_change":"2024-03-28T18:35:50.718042+0000","last_active":"2024-03-28T18:35:57.430583+0000","last_peered":"2024-03-28T18:35:57.430583+0000","last_clean":"2024-03-28T18:35:57.430583+0000","last_became_active":"2024-03-28T18:35:50.717713+0000","last_became_peered":"2024-03-28T18:35:50.717713+0000","last_unstale":"2024-03-28T18:35:57.430583+0000","last_undegraded":"2024-03-28T18:35:57.430583+0000","last_fullsized":"2024-03-28T18:35:57.430583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:51.764882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430663+0000","last_change":"2024-03-28T18:35:50.720741+0000","last_active":"2024-03-28T18:35:57.430663+0000","last_peered":"2024-03-28T18:35:57.430663+0000","last_clean":"2024-03-28T18:35:57.430663+0000","last_became_active":"2024-03-28T18:35:50.720038+0000","last_became_peered":"2024-03-28T18:35:50.720038+0000","last_unstale":"2024-03-28T18:35:57.430663+0000","last_undegraded":"2024-03-28T18:35:57.430663+0000","last_fullsized":"2024-03-28T18:35:57.430663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:48.001768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430846+0000","last_change":"2024-03-28T18:35:50.714152+0000","last_active":"2024-03-28T18:35:57.430846+0000","last_peered":"2024-03-28T18:35:57.430846+0000","last_clean":"2024-03-28T18:35:57.430846+0000","last_became_active":"2024-03-28T18:35:50.713813+0000","last_became_peered":"2024-03-28T18:35:50.713813+0000","last_unstale":"2024-03-28T18:35:57.430846+0000","last_undegraded":"2024-03-28T18:35:57.430846+0000","last_fullsized":"2024-03-28T18:35:57.430846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:48.698925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022952+0000","last_change":"2024-03-28T18:35:50.708046+0000","last_active":"2024-03-28T18:35:56.022952+0000","last_peered":"2024-03-28T18:35:56.022952+0000","last_clean":"2024-03-28T18:35:56.022952+0000","last_became_active":"2024-03-28T18:35:50.707694+0000","last_became_peered":"2024-03-28T18:35:50.707694+0000","last_unstale":"2024-03-28T18:35:56.022952+0000","last_undegraded":"2024-03-28T18:35:56.022952+0000","last_fullsized":"2024-03-28T18:35:56.022952+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:37:46.494020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023070+0000","last_change":"2024-03-28T18:35:50.714366+0000","last_active":"2024-03-28T18:35:56.023070+0000","last_peered":"2024-03-28T18:35:56.023070+0000","last_clean":"2024-03-28T18:35:56.023070+0000","last_became_active":"2024-03-28T18:35:50.714046+0000","last_became_peered":"2024-03-28T18:35:50.714046+0000","last_unstale":"2024-03-28T18:35:56.023070+0000","last_undegraded":"2024-03-28T18:35:56.023070+0000","last_fullsized":"2024-03-28T18:35:56.023070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:24.757304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022722+0000","last_change":"2024-03-28T18:35:55.032307+0000","last_active":"2024-03-28T18:35:56.022722+0000","last_peered":"2024-03-28T18:35:56.022722+0000","last_clean":"2024-03-28T18:35:56.022722+0000","last_became_active":"2024-03-28T18:35:55.031787+0000","last_became_peered":"2024-03-28T18:35:55.031787+0000","last_unstale":"2024-03-28T18:35:56.022722+0000","last_undegraded":"2024-03-28T18:35:56.022722+0000","last_fullsized":"2024-03-28T18:35:56.022722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:15.649053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023007+0000","last_change":"2024-03-28T18:35:49.933942+0000","last_active":"2024-03-28T18:35:56.023007+0000","last_peered":"2024-03-28T18:35:56.023007+0000","last_clean":"2024-03-28T18:35:56.023007+0000","last_became_active":"2024-03-28T18:35:49.933608+0000","last_became_peered":"2024-03-28T18:35:49.933608+0000","last_unstale":"2024-03-28T18:35:56.023007+0000","last_undegraded":"2024-03-28T18:35:56.023007+0000","last_fullsized":"2024-03-28T18:35:56.023007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:24.327806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380048+0000","last_change":"2024-03-28T18:35:50.704543+0000","last_active":"2024-03-28T18:35:56.380048+0000","last_peered":"2024-03-28T18:35:56.380048+0000","last_clean":"2024-03-28T18:35:56.380048+0000","last_became_active":"2024-03-28T18:35:50.704232+0000","last_became_peered":"2024-03-28T18:35:50.704232+0000","last_unstale":"2024-03-28T18:35:56.380048+0000","last_undegraded":"2024-03-28T18:35:56.380048+0000","last_fullsized":"2024-03-28T18:35:56.380048+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:30.934620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023573+0000","last_change":"2024-03-28T18:35:55.032545+0000","last_active":"2024-03-28T18:35:56.023573+0000","last_peered":"2024-03-28T18:35:56.023573+0000","last_clean":"2024-03-28T18:35:56.023573+0000","last_became_active":"2024-03-28T18:35:55.032011+0000","last_became_peered":"2024-03-28T18:35:55.032011+0000","last_unstale":"2024-03-28T18:35:56.023573+0000","last_undegraded":"2024-03-28T18:35:56.023573+0000","last_fullsized":"2024-03-28T18:35:56.023573+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:46.960816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380254+0000","last_change":"2024-03-28T18:35:49.940627+0000","last_active":"2024-03-28T18:35:56.380254+0000","last_peered":"2024-03-28T18:35:56.380254+0000","last_clean":"2024-03-28T18:35:56.380254+0000","last_became_active":"2024-03-28T18:35:49.940313+0000","last_became_peered":"2024-03-28T18:35:49.940313+0000","last_unstale":"2024-03-28T18:35:56.380254+0000","last_undegraded":"2024-03-28T18:35:56.380254+0000","last_fullsized":"2024-03-28T18:35:56.380254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:24.215713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430682+0000","last_change":"2024-03-28T18:35:50.720053+0000","last_active":"2024-03-28T18:35:57.430682+0000","last_peered":"2024-03-28T18:35:57.430682+0000","last_clean":"2024-03-28T18:35:57.430682+0000","last_became_active":"2024-03-28T18:35:50.719698+0000","last_became_peered":"2024-03-28T18:35:50.719698+0000","last_unstale":"2024-03-28T18:35:57.430682+0000","last_undegraded":"2024-03-28T18:35:57.430682+0000","last_fullsized":"2024-03-28T18:35:57.430682+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:30.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023864+0000","last_change":"2024-03-28T18:35:55.036468+0000","last_active":"2024-03-28T18:35:56.023864+0000","last_peered":"2024-03-28T18:35:56.023864+0000","last_clean":"2024-03-28T18:35:56.023864+0000","last_became_active":"2024-03-28T18:35:55.036104+0000","last_became_peered":"2024-03-28T18:35:55.036104+0000","last_unstale":"2024-03-28T18:35:56.023864+0000","last_undegraded":"2024-03-28T18:35:56.023864+0000","last_fullsized":"2024-03-28T18:35:56.023864+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:11:41.886642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.018309+0000","last_change":"2024-03-28T18:35:49.938447+0000","last_active":"2024-03-28T18:36:12.018309+0000","last_peered":"2024-03-28T18:36:12.018309+0000","last_clean":"2024-03-28T18:36:12.018309+0000","last_became_active":"2024-03-28T18:35:49.938125+0000","last_became_peered":"2024-03-28T18:35:49.938125+0000","last_unstale":"2024-03-28T18:36:12.018309+0000","last_undegraded":"2024-03-28T18:36:12.018309+0000","last_fullsized":"2024-03-28T18:36:12.018309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:31.064360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431381+0000","last_change":"2024-03-28T18:35:50.711208+0000","last_active":"2024-03-28T18:35:57.431381+0000","last_peered":"2024-03-28T18:35:57.431381+0000","last_clean":"2024-03-28T18:35:57.431381+0000","last_became_active":"2024-03-28T18:35:50.710306+0000","last_became_peered":"2024-03-28T18:35:50.710306+0000","last_unstale":"2024-03-28T18:35:57.431381+0000","last_undegraded":"2024-03-28T18:35:57.431381+0000","last_fullsized":"2024-03-28T18:35:57.431381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.449619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023605+0000","last_change":"2024-03-28T18:35:55.027968+0000","last_active":"2024-03-28T18:35:56.023605+0000","last_peered":"2024-03-28T18:35:56.023605+0000","last_clean":"2024-03-28T18:35:56.023605+0000","last_became_active":"2024-03-28T18:35:55.027486+0000","last_became_peered":"2024-03-28T18:35:55.027486+0000","last_unstale":"2024-03-28T18:35:56.023605+0000","last_undegraded":"2024-03-28T18:35:56.023605+0000","last_fullsized":"2024-03-28T18:35:56.023605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:30.088482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.017960+0000","last_change":"2024-03-28T18:35:49.936488+0000","last_active":"2024-03-28T18:36:12.017960+0000","last_peered":"2024-03-28T18:36:12.017960+0000","last_clean":"2024-03-28T18:36:12.017960+0000","last_became_active":"2024-03-28T18:35:49.936122+0000","last_became_peered":"2024-03-28T18:35:49.936122+0000","last_unstale":"2024-03-28T18:36:12.017960+0000","last_undegraded":"2024-03-28T18:36:12.017960+0000","last_fullsized":"2024-03-28T18:36:12.017960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:51.096079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431258+0000","last_change":"2024-03-28T18:35:50.704075+0000","last_active":"2024-03-28T18:35:57.431258+0000","last_peered":"2024-03-28T18:35:57.431258+0000","last_clean":"2024-03-28T18:35:57.431258+0000","last_became_active":"2024-03-28T18:35:50.703726+0000","last_became_peered":"2024-03-28T18:35:50.703726+0000","last_unstale":"2024-03-28T18:35:57.431258+0000","last_undegraded":"2024-03-28T18:35:57.431258+0000","last_fullsized":"2024-03-28T18:35:57.431258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:08.522098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023012+0000","last_change":"2024-03-28T18:35:55.023382+0000","last_active":"2024-03-28T18:35:56.023012+0000","last_peered":"2024-03-28T18:35:56.023012+0000","last_clean":"2024-03-28T18:35:56.023012+0000","last_became_active":"2024-03-28T18:35:55.022992+0000","last_became_peered":"2024-03-28T18:35:55.022992+0000","last_unstale":"2024-03-28T18:35:56.023012+0000","last_undegraded":"2024-03-28T18:35:56.023012+0000","last_fullsized":"2024-03-28T18:35:56.023012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:09.260173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430774+0000","last_change":"2024-03-28T18:35:49.939304+0000","last_active":"2024-03-28T18:35:57.430774+0000","last_peered":"2024-03-28T18:35:57.430774+0000","last_clean":"2024-03-28T18:35:57.430774+0000","last_became_active":"2024-03-28T18:35:49.938976+0000","last_became_peered":"2024-03-28T18:35:49.938976+0000","last_unstale":"2024-03-28T18:35:57.430774+0000","last_undegraded":"2024-03-28T18:35:57.430774+0000","last_fullsized":"2024-03-28T18:35:57.430774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:37.368257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030593+0000","last_change":"2024-03-28T18:35:50.715841+0000","last_active":"2024-03-28T18:35:57.030593+0000","last_peered":"2024-03-28T18:35:57.030593+0000","last_clean":"2024-03-28T18:35:57.030593+0000","last_became_active":"2024-03-28T18:35:50.715365+0000","last_became_peered":"2024-03-28T18:35:50.715365+0000","last_unstale":"2024-03-28T18:35:57.030593+0000","last_undegraded":"2024-03-28T18:35:57.030593+0000","last_fullsized":"2024-03-28T18:35:57.030593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:33.009378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023001+0000","last_change":"2024-03-28T18:35:55.033073+0000","last_active":"2024-03-28T18:35:56.023001+0000","last_peered":"2024-03-28T18:35:56.023001+0000","last_clean":"2024-03-28T18:35:56.023001+0000","last_became_active":"2024-03-28T18:35:55.032649+0000","last_became_peered":"2024-03-28T18:35:55.032649+0000","last_unstale":"2024-03-28T18:35:56.023001+0000","last_undegraded":"2024-03-28T18:35:56.023001+0000","last_fullsized":"2024-03-28T18:35:56.023001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:14.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:10.091208+0000","last_change":"2024-03-28T18:35:49.932672+0000","last_active":"2024-03-28T18:36:10.091208+0000","last_peered":"2024-03-28T18:36:10.091208+0000","last_clean":"2024-03-28T18:36:10.091208+0000","last_became_active":"2024-03-28T18:35:49.932207+0000","last_became_peered":"2024-03-28T18:35:49.932207+0000","last_unstale":"2024-03-28T18:36:10.091208+0000","last_undegraded":"2024-03-28T18:36:10.091208+0000","last_fullsized":"2024-03-28T18:36:10.091208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:50.606075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430902+0000","last_change":"2024-03-28T18:35:50.696535+0000","last_active":"2024-03-28T18:35:57.430902+0000","last_peered":"2024-03-28T18:35:57.430902+0000","last_clean":"2024-03-28T18:35:57.430902+0000","last_became_active":"2024-03-28T18:35:50.696142+0000","last_became_peered":"2024-03-28T18:35:50.696142+0000","last_unstale":"2024-03-28T18:35:57.430902+0000","last_undegraded":"2024-03-28T18:35:57.430902+0000","last_fullsized":"2024-03-28T18:35:57.430902+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:24:59.749302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430866+0000","last_change":"2024-03-28T18:35:55.032203+0000","last_active":"2024-03-28T18:35:57.430866+0000","last_peered":"2024-03-28T18:35:57.430866+0000","last_clean":"2024-03-28T18:35:57.430866+0000","last_became_active":"2024-03-28T18:35:55.031734+0000","last_became_peered":"2024-03-28T18:35:55.031734+0000","last_unstale":"2024-03-28T18:35:57.430866+0000","last_undegraded":"2024-03-28T18:35:57.430866+0000","last_fullsized":"2024-03-28T18:35:57.430866+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:11:08.224595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:10.092750+0000","last_change":"2024-03-28T18:35:49.929798+0000","last_active":"2024-03-28T18:36:10.092750+0000","last_peered":"2024-03-28T18:36:10.092750+0000","last_clean":"2024-03-28T18:36:10.092750+0000","last_became_active":"2024-03-28T18:35:49.929278+0000","last_became_peered":"2024-03-28T18:35:49.929278+0000","last_unstale":"2024-03-28T18:36:10.092750+0000","last_undegraded":"2024-03-28T18:36:10.092750+0000","last_fullsized":"2024-03-28T18:36:10.092750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:33:13.078225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379574+0000","last_change":"2024-03-28T18:35:50.702935+0000","last_active":"2024-03-28T18:35:56.379574+0000","last_peered":"2024-03-28T18:35:56.379574+0000","last_clean":"2024-03-28T18:35:56.379574+0000","last_became_active":"2024-03-28T18:35:50.702586+0000","last_became_peered":"2024-03-28T18:35:50.702586+0000","last_unstale":"2024-03-28T18:35:56.379574+0000","last_undegraded":"2024-03-28T18:35:56.379574+0000","last_fullsized":"2024-03-28T18:35:56.379574+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:50.335605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430722+0000","last_change":"2024-03-28T18:35:55.028541+0000","last_active":"2024-03-28T18:35:57.430722+0000","last_peered":"2024-03-28T18:35:57.430722+0000","last_clean":"2024-03-28T18:35:57.430722+0000","last_became_active":"2024-03-28T18:35:55.028142+0000","last_became_peered":"2024-03-28T18:35:55.028142+0000","last_unstale":"2024-03-28T18:35:57.430722+0000","last_undegraded":"2024-03-28T18:35:57.430722+0000","last_fullsized":"2024-03-28T18:35:57.430722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:20.738071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430486+0000","last_change":"2024-03-28T18:35:49.938255+0000","last_active":"2024-03-28T18:35:57.430486+0000","last_peered":"2024-03-28T18:35:57.430486+0000","last_clean":"2024-03-28T18:35:57.430486+0000","last_became_active":"2024-03-28T18:35:49.937930+0000","last_became_peered":"2024-03-28T18:35:49.937930+0000","last_unstale":"2024-03-28T18:35:57.430486+0000","last_undegraded":"2024-03-28T18:35:57.430486+0000","last_fullsized":"2024-03-28T18:35:57.430486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:41.621941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022472+0000","last_change":"2024-03-28T18:35:50.706437+0000","last_active":"2024-03-28T18:35:56.022472+0000","last_peered":"2024-03-28T18:35:56.022472+0000","last_clean":"2024-03-28T18:35:56.022472+0000","last_became_active":"2024-03-28T18:35:50.706139+0000","last_became_peered":"2024-03-28T18:35:50.706139+0000","last_unstale":"2024-03-28T18:35:56.022472+0000","last_undegraded":"2024-03-28T18:35:56.022472+0000","last_fullsized":"2024-03-28T18:35:56.022472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:52.750896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379906+0000","last_change":"2024-03-28T18:35:55.025289+0000","last_active":"2024-03-28T18:35:56.379906+0000","last_peered":"2024-03-28T18:35:56.379906+0000","last_clean":"2024-03-28T18:35:56.379906+0000","last_became_active":"2024-03-28T18:35:55.024835+0000","last_became_peered":"2024-03-28T18:35:55.024835+0000","last_unstale":"2024-03-28T18:35:56.379906+0000","last_undegraded":"2024-03-28T18:35:56.379906+0000","last_fullsized":"2024-03-28T18:35:56.379906+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:08.933559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.080816+0000","last_change":"2024-03-28T18:35:49.940518+0000","last_active":"2024-03-28T18:36:09.080816+0000","last_peered":"2024-03-28T18:36:09.080816+0000","last_clean":"2024-03-28T18:36:09.080816+0000","last_became_active":"2024-03-28T18:35:49.940208+0000","last_became_peered":"2024-03-28T18:35:49.940208+0000","last_unstale":"2024-03-28T18:36:09.080816+0000","last_undegraded":"2024-03-28T18:36:09.080816+0000","last_fullsized":"2024-03-28T18:36:09.080816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:55.557840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431590+0000","last_change":"2024-03-28T18:35:50.710781+0000","last_active":"2024-03-28T18:35:57.431590+0000","last_peered":"2024-03-28T18:35:57.431590+0000","last_clean":"2024-03-28T18:35:57.431590+0000","last_became_active":"2024-03-28T18:35:50.710458+0000","last_became_peered":"2024-03-28T18:35:50.710458+0000","last_unstale":"2024-03-28T18:35:57.431590+0000","last_undegraded":"2024-03-28T18:35:57.431590+0000","last_fullsized":"2024-03-28T18:35:57.431590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:49.020331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022622+0000","last_change":"2024-03-28T18:35:55.034023+0000","last_active":"2024-03-28T18:35:56.022622+0000","last_peered":"2024-03-28T18:35:56.022622+0000","last_clean":"2024-03-28T18:35:56.022622+0000","last_became_active":"2024-03-28T18:35:55.033634+0000","last_became_peered":"2024-03-28T18:35:55.033634+0000","last_unstale":"2024-03-28T18:35:56.022622+0000","last_undegraded":"2024-03-28T18:35:56.022622+0000","last_fullsized":"2024-03-28T18:35:56.022622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:41.221061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431709+0000","last_change":"2024-03-28T18:35:49.937443+0000","last_active":"2024-03-28T18:35:57.431709+0000","last_peered":"2024-03-28T18:35:57.431709+0000","last_clean":"2024-03-28T18:35:57.431709+0000","last_became_active":"2024-03-28T18:35:49.937003+0000","last_became_peered":"2024-03-28T18:35:49.937003+0000","last_unstale":"2024-03-28T18:35:57.431709+0000","last_undegraded":"2024-03-28T18:35:57.431709+0000","last_fullsized":"2024-03-28T18:35:57.431709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:49:05.804868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430721+0000","last_change":"2024-03-28T18:35:50.718480+0000","last_active":"2024-03-28T18:35:57.430721+0000","last_peered":"2024-03-28T18:35:57.430721+0000","last_clean":"2024-03-28T18:35:57.430721+0000","last_became_active":"2024-03-28T18:35:50.717253+0000","last_became_peered":"2024-03-28T18:35:50.717253+0000","last_unstale":"2024-03-28T18:35:57.430721+0000","last_undegraded":"2024-03-28T18:35:57.430721+0000","last_fullsized":"2024-03-28T18:35:57.430721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:59:30.071806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024149+0000","last_change":"2024-03-28T18:35:55.032461+0000","last_active":"2024-03-28T18:35:56.024149+0000","last_peered":"2024-03-28T18:35:56.024149+0000","last_clean":"2024-03-28T18:35:56.024149+0000","last_became_active":"2024-03-28T18:35:55.032029+0000","last_became_peered":"2024-03-28T18:35:55.032029+0000","last_unstale":"2024-03-28T18:35:56.024149+0000","last_undegraded":"2024-03-28T18:35:56.024149+0000","last_fullsized":"2024-03-28T18:35:56.024149+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:27:11.216371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431223+0000","last_change":"2024-03-28T18:35:49.934357+0000","last_active":"2024-03-28T18:35:57.431223+0000","last_peered":"2024-03-28T18:35:57.431223+0000","last_clean":"2024-03-28T18:35:57.431223+0000","last_became_active":"2024-03-28T18:35:49.933994+0000","last_became_peered":"2024-03-28T18:35:49.933994+0000","last_unstale":"2024-03-28T18:35:57.431223+0000","last_undegraded":"2024-03-28T18:35:57.431223+0000","last_fullsized":"2024-03-28T18:35:57.431223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:21.408585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431532+0000","last_change":"2024-03-28T18:35:50.716771+0000","last_active":"2024-03-28T18:35:57.431532+0000","last_peered":"2024-03-28T18:35:57.431532+0000","last_clean":"2024-03-28T18:35:57.431532+0000","last_became_active":"2024-03-28T18:35:50.716393+0000","last_became_peered":"2024-03-28T18:35:50.716393+0000","last_unstale":"2024-03-28T18:35:57.431532+0000","last_undegraded":"2024-03-28T18:35:57.431532+0000","last_fullsized":"2024-03-28T18:35:57.431532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:33:10.881475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022479+0000","last_change":"2024-03-28T18:35:55.027650+0000","last_active":"2024-03-28T18:35:56.022479+0000","last_peered":"2024-03-28T18:35:56.022479+0000","last_clean":"2024-03-28T18:35:56.022479+0000","last_became_active":"2024-03-28T18:35:55.027109+0000","last_became_peered":"2024-03-28T18:35:55.027109+0000","last_unstale":"2024-03-28T18:35:56.022479+0000","last_undegraded":"2024-03-28T18:35:56.022479+0000","last_fullsized":"2024-03-28T18:35:56.022479+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:54.826415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.083443+0000","last_change":"2024-03-28T18:35:49.934499+0000","last_active":"2024-03-28T18:36:09.083443+0000","last_peered":"2024-03-28T18:36:09.083443+0000","last_clean":"2024-03-28T18:36:09.083443+0000","last_became_active":"2024-03-28T18:35:49.934150+0000","last_became_peered":"2024-03-28T18:35:49.934150+0000","last_unstale":"2024-03-28T18:36:09.083443+0000","last_undegraded":"2024-03-28T18:36:09.083443+0000","last_fullsized":"2024-03-28T18:36:09.083443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:53.661593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030694+0000","last_change":"2024-03-28T18:35:50.714162+0000","last_active":"2024-03-28T18:35:57.030694+0000","last_peered":"2024-03-28T18:35:57.030694+0000","last_clean":"2024-03-28T18:35:57.030694+0000","last_became_active":"2024-03-28T18:35:50.713819+0000","last_became_peered":"2024-03-28T18:35:50.713819+0000","last_unstale":"2024-03-28T18:35:57.030694+0000","last_undegraded":"2024-03-28T18:35:57.030694+0000","last_fullsized":"2024-03-28T18:35:57.030694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:49:19.365559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431591+0000","last_change":"2024-03-28T18:35:55.028460+0000","last_active":"2024-03-28T18:35:57.431591+0000","last_peered":"2024-03-28T18:35:57.431591+0000","last_clean":"2024-03-28T18:35:57.431591+0000","last_became_active":"2024-03-28T18:35:55.028015+0000","last_became_peered":"2024-03-28T18:35:55.028015+0000","last_unstale":"2024-03-28T18:35:57.431591+0000","last_undegraded":"2024-03-28T18:35:57.431591+0000","last_fullsized":"2024-03-28T18:35:57.431591+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:00.864611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022804+0000","last_change":"2024-03-28T18:35:49.938090+0000","last_active":"2024-03-28T18:35:56.022804+0000","last_peered":"2024-03-28T18:35:56.022804+0000","last_clean":"2024-03-28T18:35:56.022804+0000","last_became_active":"2024-03-28T18:35:49.937572+0000","last_became_peered":"2024-03-28T18:35:49.937572+0000","last_unstale":"2024-03-28T18:35:56.022804+0000","last_undegraded":"2024-03-28T18:35:56.022804+0000","last_fullsized":"2024-03-28T18:35:56.022804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:41.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179442+0000","last_change":"2024-03-28T18:35:50.717286+0000","last_active":"2024-03-28T18:35:56.179442+0000","last_peered":"2024-03-28T18:35:56.179442+0000","last_clean":"2024-03-28T18:35:56.179442+0000","last_became_active":"2024-03-28T18:35:50.712917+0000","last_became_peered":"2024-03-28T18:35:50.712917+0000","last_unstale":"2024-03-28T18:35:56.179442+0000","last_undegraded":"2024-03-28T18:35:56.179442+0000","last_fullsized":"2024-03-28T18:35:56.179442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:10:42.306028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430732+0000","last_change":"2024-03-28T18:35:55.023384+0000","last_active":"2024-03-28T18:35:57.430732+0000","last_peered":"2024-03-28T18:35:57.430732+0000","last_clean":"2024-03-28T18:35:57.430732+0000","last_became_active":"2024-03-28T18:35:55.023013+0000","last_became_peered":"2024-03-28T18:35:55.023013+0000","last_unstale":"2024-03-28T18:35:57.430732+0000","last_undegraded":"2024-03-28T18:35:57.430732+0000","last_fullsized":"2024-03-28T18:35:57.430732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:01.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431150+0000","last_change":"2024-03-28T18:35:49.939073+0000","last_active":"2024-03-28T18:35:57.431150+0000","last_peered":"2024-03-28T18:35:57.431150+0000","last_clean":"2024-03-28T18:35:57.431150+0000","last_became_active":"2024-03-28T18:35:49.938764+0000","last_became_peered":"2024-03-28T18:35:49.938764+0000","last_unstale":"2024-03-28T18:35:57.431150+0000","last_undegraded":"2024-03-28T18:35:57.431150+0000","last_fullsized":"2024-03-28T18:35:57.431150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:52.089059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430987+0000","last_change":"2024-03-28T18:35:50.716132+0000","last_active":"2024-03-28T18:35:57.430987+0000","last_peered":"2024-03-28T18:35:57.430987+0000","last_clean":"2024-03-28T18:35:57.430987+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:35:57.430987+0000","last_undegraded":"2024-03-28T18:35:57.430987+0000","last_fullsized":"2024-03-28T18:35:57.430987+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:57.930739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022880+0000","last_change":"2024-03-28T18:35:55.035553+0000","last_active":"2024-03-28T18:35:56.022880+0000","last_peered":"2024-03-28T18:35:56.022880+0000","last_clean":"2024-03-28T18:35:56.022880+0000","last_became_active":"2024-03-28T18:35:55.035134+0000","last_became_peered":"2024-03-28T18:35:55.035134+0000","last_unstale":"2024-03-28T18:35:56.022880+0000","last_undegraded":"2024-03-28T18:35:56.022880+0000","last_fullsized":"2024-03-28T18:35:56.022880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:03:52.976361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023453+0000","last_change":"2024-03-28T18:35:49.932696+0000","last_active":"2024-03-28T18:35:56.023453+0000","last_peered":"2024-03-28T18:35:56.023453+0000","last_clean":"2024-03-28T18:35:56.023453+0000","last_became_active":"2024-03-28T18:35:49.932350+0000","last_became_peered":"2024-03-28T18:35:49.932350+0000","last_unstale":"2024-03-28T18:35:56.023453+0000","last_undegraded":"2024-03-28T18:35:56.023453+0000","last_fullsized":"2024-03-28T18:35:56.023453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:26.183549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430791+0000","last_change":"2024-03-28T18:35:50.712734+0000","last_active":"2024-03-28T18:35:57.430791+0000","last_peered":"2024-03-28T18:35:57.430791+0000","last_clean":"2024-03-28T18:35:57.430791+0000","last_became_active":"2024-03-28T18:35:50.712409+0000","last_became_peered":"2024-03-28T18:35:50.712409+0000","last_unstale":"2024-03-28T18:35:57.430791+0000","last_undegraded":"2024-03-28T18:35:57.430791+0000","last_fullsized":"2024-03-28T18:35:57.430791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:36.344182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179901+0000","last_change":"2024-03-28T18:35:55.025047+0000","last_active":"2024-03-28T18:35:56.179901+0000","last_peered":"2024-03-28T18:35:56.179901+0000","last_clean":"2024-03-28T18:35:56.179901+0000","last_became_active":"2024-03-28T18:35:55.024652+0000","last_became_peered":"2024-03-28T18:35:55.024652+0000","last_unstale":"2024-03-28T18:35:56.179901+0000","last_undegraded":"2024-03-28T18:35:56.179901+0000","last_fullsized":"2024-03-28T18:35:56.179901+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:17.750951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.008930+0000","last_change":"2024-03-28T18:35:49.940506+0000","last_active":"2024-03-28T18:36:12.008930+0000","last_peered":"2024-03-28T18:36:12.008930+0000","last_clean":"2024-03-28T18:36:12.008930+0000","last_became_active":"2024-03-28T18:35:49.940168+0000","last_became_peered":"2024-03-28T18:35:49.940168+0000","last_unstale":"2024-03-28T18:36:12.008930+0000","last_undegraded":"2024-03-28T18:36:12.008930+0000","last_fullsized":"2024-03-28T18:36:12.008930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:14:47.629255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430825+0000","last_change":"2024-03-28T18:35:50.705023+0000","last_active":"2024-03-28T18:35:57.430825+0000","last_peered":"2024-03-28T18:35:57.430825+0000","last_clean":"2024-03-28T18:35:57.430825+0000","last_became_active":"2024-03-28T18:35:50.704694+0000","last_became_peered":"2024-03-28T18:35:50.704694+0000","last_unstale":"2024-03-28T18:35:57.430825+0000","last_undegraded":"2024-03-28T18:35:57.430825+0000","last_fullsized":"2024-03-28T18:35:57.430825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:46.677189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430770+0000","last_change":"2024-03-28T18:35:55.028311+0000","last_active":"2024-03-28T18:35:57.430770+0000","last_peered":"2024-03-28T18:35:57.430770+0000","last_clean":"2024-03-28T18:35:57.430770+0000","last_became_active":"2024-03-28T18:35:55.027914+0000","last_became_peered":"2024-03-28T18:35:55.027914+0000","last_unstale":"2024-03-28T18:35:57.430770+0000","last_undegraded":"2024-03-28T18:35:57.430770+0000","last_fullsized":"2024-03-28T18:35:57.430770+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:59.684230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:11.086917+0000","last_change":"2024-03-28T18:35:49.938004+0000","last_active":"2024-03-28T18:36:11.086917+0000","last_peered":"2024-03-28T18:36:11.086917+0000","last_clean":"2024-03-28T18:36:11.086917+0000","last_became_active":"2024-03-28T18:35:49.937615+0000","last_became_peered":"2024-03-28T18:35:49.937615+0000","last_unstale":"2024-03-28T18:36:11.086917+0000","last_undegraded":"2024-03-28T18:36:11.086917+0000","last_fullsized":"2024-03-28T18:36:11.086917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:21.619857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430623+0000","last_change":"2024-03-28T18:35:50.719276+0000","last_active":"2024-03-28T18:35:57.430623+0000","last_peered":"2024-03-28T18:35:57.430623+0000","last_clean":"2024-03-28T18:35:57.430623+0000","last_became_active":"2024-03-28T18:35:50.718896+0000","last_became_peered":"2024-03-28T18:35:50.718896+0000","last_unstale":"2024-03-28T18:35:57.430623+0000","last_undegraded":"2024-03-28T18:35:57.430623+0000","last_fullsized":"2024-03-28T18:35:57.430623+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:24.494713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030708+0000","last_change":"2024-03-28T18:35:55.099193+0000","last_active":"2024-03-28T18:35:57.030708+0000","last_peered":"2024-03-28T18:35:57.030708+0000","last_clean":"2024-03-28T18:35:57.030708+0000","last_became_active":"2024-03-28T18:35:55.098795+0000","last_became_peered":"2024-03-28T18:35:55.098795+0000","last_unstale":"2024-03-28T18:35:57.030708+0000","last_undegraded":"2024-03-28T18:35:57.030708+0000","last_fullsized":"2024-03-28T18:35:57.030708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:07.872554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022664+0000","last_change":"2024-03-28T18:35:49.936885+0000","last_active":"2024-03-28T18:35:56.022664+0000","last_peered":"2024-03-28T18:35:56.022664+0000","last_clean":"2024-03-28T18:35:56.022664+0000","last_became_active":"2024-03-28T18:35:49.936563+0000","last_became_peered":"2024-03-28T18:35:49.936563+0000","last_unstale":"2024-03-28T18:35:56.022664+0000","last_undegraded":"2024-03-28T18:35:56.022664+0000","last_fullsized":"2024-03-28T18:35:56.022664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:33.631160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794247+0000","last_change":"2024-03-28T18:35:50.713750+0000","last_active":"2024-03-28T18:35:56.794247+0000","last_peered":"2024-03-28T18:35:56.794247+0000","last_clean":"2024-03-28T18:35:56.794247+0000","last_became_active":"2024-03-28T18:35:50.713378+0000","last_became_peered":"2024-03-28T18:35:50.713378+0000","last_unstale":"2024-03-28T18:35:56.794247+0000","last_undegraded":"2024-03-28T18:35:56.794247+0000","last_fullsized":"2024-03-28T18:35:56.794247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:36.072651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179597+0000","last_change":"2024-03-28T18:35:55.030827+0000","last_active":"2024-03-28T18:35:56.179597+0000","last_peered":"2024-03-28T18:35:56.179597+0000","last_clean":"2024-03-28T18:35:56.179597+0000","last_became_active":"2024-03-28T18:35:55.030382+0000","last_became_peered":"2024-03-28T18:35:55.030382+0000","last_unstale":"2024-03-28T18:35:56.179597+0000","last_undegraded":"2024-03-28T18:35:56.179597+0000","last_fullsized":"2024-03-28T18:35:56.179597+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:41:58.371568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431032+0000","last_change":"2024-03-28T18:35:49.932290+0000","last_active":"2024-03-28T18:35:57.431032+0000","last_peered":"2024-03-28T18:35:57.431032+0000","last_clean":"2024-03-28T18:35:57.431032+0000","last_became_active":"2024-03-28T18:35:49.931925+0000","last_became_peered":"2024-03-28T18:35:49.931925+0000","last_unstale":"2024-03-28T18:35:57.431032+0000","last_undegraded":"2024-03-28T18:35:57.431032+0000","last_fullsized":"2024-03-28T18:35:57.431032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:26.477117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.794044+0000","last_change":"2024-03-28T18:35:50.713654+0000","last_active":"2024-03-28T18:35:56.794044+0000","last_peered":"2024-03-28T18:35:56.794044+0000","last_clean":"2024-03-28T18:35:56.794044+0000","last_became_active":"2024-03-28T18:35:50.713146+0000","last_became_peered":"2024-03-28T18:35:50.713146+0000","last_unstale":"2024-03-28T18:35:56.794044+0000","last_undegraded":"2024-03-28T18:35:56.794044+0000","last_fullsized":"2024-03-28T18:35:56.794044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:23.951963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179820+0000","last_change":"2024-03-28T18:35:55.099331+0000","last_active":"2024-03-28T18:35:56.179820+0000","last_peered":"2024-03-28T18:35:56.179820+0000","last_clean":"2024-03-28T18:35:56.179820+0000","last_became_active":"2024-03-28T18:35:55.098912+0000","last_became_peered":"2024-03-28T18:35:55.098912+0000","last_unstale":"2024-03-28T18:35:56.179820+0000","last_undegraded":"2024-03-28T18:35:56.179820+0000","last_fullsized":"2024-03-28T18:35:56.179820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:31.082148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430700+0000","last_change":"2024-03-28T18:35:50.704377+0000","last_active":"2024-03-28T18:35:57.430700+0000","last_peered":"2024-03-28T18:35:57.430700+0000","last_clean":"2024-03-28T18:35:57.430700+0000","last_became_active":"2024-03-28T18:35:50.703991+0000","last_became_peered":"2024-03-28T18:35:50.703991+0000","last_unstale":"2024-03-28T18:35:57.430700+0000","last_undegraded":"2024-03-28T18:35:57.430700+0000","last_fullsized":"2024-03-28T18:35:57.430700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:48.173466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015411+0000","last_change":"2024-03-28T18:35:49.933528+0000","last_active":"2024-03-28T18:36:12.015411+0000","last_peered":"2024-03-28T18:36:12.015411+0000","last_clean":"2024-03-28T18:36:12.015411+0000","last_became_active":"2024-03-28T18:35:49.933171+0000","last_became_peered":"2024-03-28T18:35:49.933171+0000","last_unstale":"2024-03-28T18:36:12.015411+0000","last_undegraded":"2024-03-28T18:36:12.015411+0000","last_fullsized":"2024-03-28T18:36:12.015411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:50:59.928400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431778+0000","last_change":"2024-03-28T18:35:55.029752+0000","last_active":"2024-03-28T18:35:57.431778+0000","last_peered":"2024-03-28T18:35:57.431778+0000","last_clean":"2024-03-28T18:35:57.431778+0000","last_became_active":"2024-03-28T18:35:55.029407+0000","last_became_peered":"2024-03-28T18:35:55.029407+0000","last_unstale":"2024-03-28T18:35:57.431778+0000","last_undegraded":"2024-03-28T18:35:57.431778+0000","last_fullsized":"2024-03-28T18:35:57.431778+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:17.328315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379713+0000","last_change":"2024-03-28T18:35:50.706380+0000","last_active":"2024-03-28T18:35:56.379713+0000","last_peered":"2024-03-28T18:35:56.379713+0000","last_clean":"2024-03-28T18:35:56.379713+0000","last_became_active":"2024-03-28T18:35:50.706044+0000","last_became_peered":"2024-03-28T18:35:50.706044+0000","last_unstale":"2024-03-28T18:35:56.379713+0000","last_undegraded":"2024-03-28T18:35:56.379713+0000","last_fullsized":"2024-03-28T18:35:56.379713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:04.916646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023655+0000","last_change":"2024-03-28T18:35:49.930241+0000","last_active":"2024-03-28T18:35:56.023655+0000","last_peered":"2024-03-28T18:35:56.023655+0000","last_clean":"2024-03-28T18:35:56.023655+0000","last_became_active":"2024-03-28T18:35:49.929830+0000","last_became_peered":"2024-03-28T18:35:49.929830+0000","last_unstale":"2024-03-28T18:35:56.023655+0000","last_undegraded":"2024-03-28T18:35:56.023655+0000","last_fullsized":"2024-03-28T18:35:56.023655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:11.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431044+0000","last_change":"2024-03-28T18:34:38.804796+0000","last_active":"2024-03-28T18:35:57.431044+0000","last_peered":"2024-03-28T18:35:57.431044+0000","last_clean":"2024-03-28T18:35:57.431044+0000","last_became_active":"2024-03-28T18:34:38.804419+0000","last_became_peered":"2024-03-28T18:34:38.804419+0000","last_unstale":"2024-03-28T18:35:57.431044+0000","last_undegraded":"2024-03-28T18:35:57.431044+0000","last_fullsized":"2024-03-28T18:35:57.431044+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_clean_scrub_stamp":"2024-03-28T18:31:13.733610+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:38.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431487+0000","last_change":"2024-03-28T18:35:55.099390+0000","last_active":"2024-03-28T18:35:57.431487+0000","last_peered":"2024-03-28T18:35:57.431487+0000","last_clean":"2024-03-28T18:35:57.431487+0000","last_became_active":"2024-03-28T18:35:55.098960+0000","last_became_peered":"2024-03-28T18:35:55.098960+0000","last_unstale":"2024-03-28T18:35:57.431487+0000","last_undegraded":"2024-03-28T18:35:57.431487+0000","last_fullsized":"2024-03-28T18:35:57.431487+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:22.460846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179226+0000","last_change":"2024-03-28T18:35:50.711961+0000","last_active":"2024-03-28T18:35:56.179226+0000","last_peered":"2024-03-28T18:35:56.179226+0000","last_clean":"2024-03-28T18:35:56.179226+0000","last_became_active":"2024-03-28T18:35:50.705466+0000","last_became_peered":"2024-03-28T18:35:50.705466+0000","last_unstale":"2024-03-28T18:35:56.179226+0000","last_undegraded":"2024-03-28T18:35:56.179226+0000","last_fullsized":"2024-03-28T18:35:56.179226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:02.527220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430673+0000","last_change":"2024-03-28T18:35:49.936938+0000","last_active":"2024-03-28T18:35:57.430673+0000","last_peered":"2024-03-28T18:35:57.430673+0000","last_clean":"2024-03-28T18:35:57.430673+0000","last_became_active":"2024-03-28T18:35:49.936562+0000","last_became_peered":"2024-03-28T18:35:49.936562+0000","last_unstale":"2024-03-28T18:35:57.430673+0000","last_undegraded":"2024-03-28T18:35:57.430673+0000","last_fullsized":"2024-03-28T18:35:57.430673+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:18.067381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430893+0000","last_change":"2024-03-28T18:35:55.023402+0000","last_active":"2024-03-28T18:35:57.430893+0000","last_peered":"2024-03-28T18:35:57.430893+0000","last_clean":"2024-03-28T18:35:57.430893+0000","last_became_active":"2024-03-28T18:35:55.023006+0000","last_became_peered":"2024-03-28T18:35:55.023006+0000","last_unstale":"2024-03-28T18:35:57.430893+0000","last_undegraded":"2024-03-28T18:35:57.430893+0000","last_fullsized":"2024-03-28T18:35:57.430893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:12.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.008832+0000","last_change":"2024-03-28T18:35:49.942494+0000","last_active":"2024-03-28T18:36:12.008832+0000","last_peered":"2024-03-28T18:36:12.008832+0000","last_clean":"2024-03-28T18:36:12.008832+0000","last_became_active":"2024-03-28T18:35:49.941607+0000","last_became_peered":"2024-03-28T18:35:49.941607+0000","last_unstale":"2024-03-28T18:36:12.008832+0000","last_undegraded":"2024-03-28T18:36:12.008832+0000","last_fullsized":"2024-03-28T18:36:12.008832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:25.315100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179646+0000","last_change":"2024-03-28T18:35:50.707912+0000","last_active":"2024-03-28T18:35:56.179646+0000","last_peered":"2024-03-28T18:35:56.179646+0000","last_clean":"2024-03-28T18:35:56.179646+0000","last_became_active":"2024-03-28T18:35:50.707304+0000","last_became_peered":"2024-03-28T18:35:50.707304+0000","last_unstale":"2024-03-28T18:35:56.179646+0000","last_undegraded":"2024-03-28T18:35:56.179646+0000","last_fullsized":"2024-03-28T18:35:56.179646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:45.556370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379711+0000","last_change":"2024-03-28T18:35:55.099919+0000","last_active":"2024-03-28T18:35:56.379711+0000","last_peered":"2024-03-28T18:35:56.379711+0000","last_clean":"2024-03-28T18:35:56.379711+0000","last_became_active":"2024-03-28T18:35:55.099430+0000","last_became_peered":"2024-03-28T18:35:55.099430+0000","last_unstale":"2024-03-28T18:35:56.379711+0000","last_undegraded":"2024-03-28T18:35:56.379711+0000","last_fullsized":"2024-03-28T18:35:56.379711+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:30.091473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022950+0000","last_change":"2024-03-28T18:35:49.935871+0000","last_active":"2024-03-28T18:35:56.022950+0000","last_peered":"2024-03-28T18:35:56.022950+0000","last_clean":"2024-03-28T18:35:56.022950+0000","last_became_active":"2024-03-28T18:35:49.935536+0000","last_became_peered":"2024-03-28T18:35:49.935536+0000","last_unstale":"2024-03-28T18:35:56.022950+0000","last_undegraded":"2024-03-28T18:35:56.022950+0000","last_fullsized":"2024-03-28T18:35:56.022950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:22.915067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380161+0000","last_change":"2024-03-28T18:35:50.710877+0000","last_active":"2024-03-28T18:35:56.380161+0000","last_peered":"2024-03-28T18:35:56.380161+0000","last_clean":"2024-03-28T18:35:56.380161+0000","last_became_active":"2024-03-28T18:35:50.709987+0000","last_became_peered":"2024-03-28T18:35:50.709987+0000","last_unstale":"2024-03-28T18:35:56.380161+0000","last_undegraded":"2024-03-28T18:35:56.380161+0000","last_fullsized":"2024-03-28T18:35:56.380161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:43.092541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022648+0000","last_change":"2024-03-28T18:35:55.023390+0000","last_active":"2024-03-28T18:35:56.022648+0000","last_peered":"2024-03-28T18:35:56.022648+0000","last_clean":"2024-03-28T18:35:56.022648+0000","last_became_active":"2024-03-28T18:35:55.023034+0000","last_became_peered":"2024-03-28T18:35:55.023034+0000","last_unstale":"2024-03-28T18:35:56.022648+0000","last_undegraded":"2024-03-28T18:35:56.022648+0000","last_fullsized":"2024-03-28T18:35:56.022648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:05.829144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024232+0000","last_change":"2024-03-28T18:35:49.932675+0000","last_active":"2024-03-28T18:35:56.024232+0000","last_peered":"2024-03-28T18:35:56.024232+0000","last_clean":"2024-03-28T18:35:56.024232+0000","last_became_active":"2024-03-28T18:35:49.932246+0000","last_became_peered":"2024-03-28T18:35:49.932246+0000","last_unstale":"2024-03-28T18:35:56.024232+0000","last_undegraded":"2024-03-28T18:35:56.024232+0000","last_fullsized":"2024-03-28T18:35:56.024232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:46.806411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430754+0000","last_change":"2024-03-28T18:35:50.716861+0000","last_active":"2024-03-28T18:35:57.430754+0000","last_peered":"2024-03-28T18:35:57.430754+0000","last_clean":"2024-03-28T18:35:57.430754+0000","last_became_active":"2024-03-28T18:35:50.716377+0000","last_became_peered":"2024-03-28T18:35:50.716377+0000","last_unstale":"2024-03-28T18:35:57.430754+0000","last_undegraded":"2024-03-28T18:35:57.430754+0000","last_fullsized":"2024-03-28T18:35:57.430754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:27.018809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023989+0000","last_change":"2024-03-28T18:35:55.099891+0000","last_active":"2024-03-28T18:35:56.023989+0000","last_peered":"2024-03-28T18:35:56.023989+0000","last_clean":"2024-03-28T18:35:56.023989+0000","last_became_active":"2024-03-28T18:35:55.098988+0000","last_became_peered":"2024-03-28T18:35:55.098988+0000","last_unstale":"2024-03-28T18:35:56.023989+0000","last_undegraded":"2024-03-28T18:35:56.023989+0000","last_fullsized":"2024-03-28T18:35:56.023989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:15.682731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.082560+0000","last_change":"2024-03-28T18:35:49.936146+0000","last_active":"2024-03-28T18:36:09.082560+0000","last_peered":"2024-03-28T18:36:09.082560+0000","last_clean":"2024-03-28T18:36:09.082560+0000","last_became_active":"2024-03-28T18:35:49.935771+0000","last_became_peered":"2024-03-28T18:35:49.935771+0000","last_unstale":"2024-03-28T18:36:09.082560+0000","last_undegraded":"2024-03-28T18:36:09.082560+0000","last_fullsized":"2024-03-28T18:36:09.082560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:58.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.030376+0000","last_change":"2024-03-28T18:35:50.707527+0000","last_active":"2024-03-28T18:35:57.030376+0000","last_peered":"2024-03-28T18:35:57.030376+0000","last_clean":"2024-03-28T18:35:57.030376+0000","last_became_active":"2024-03-28T18:35:50.707216+0000","last_became_peered":"2024-03-28T18:35:50.707216+0000","last_unstale":"2024-03-28T18:35:57.030376+0000","last_undegraded":"2024-03-28T18:35:57.030376+0000","last_fullsized":"2024-03-28T18:35:57.030376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:32.344898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022955+0000","last_change":"2024-03-28T18:35:55.033294+0000","last_active":"2024-03-28T18:35:56.022955+0000","last_peered":"2024-03-28T18:35:56.022955+0000","last_clean":"2024-03-28T18:35:56.022955+0000","last_became_active":"2024-03-28T18:35:55.030743+0000","last_became_peered":"2024-03-28T18:35:55.030743+0000","last_unstale":"2024-03-28T18:35:56.022955+0000","last_undegraded":"2024-03-28T18:35:56.022955+0000","last_fullsized":"2024-03-28T18:35:56.022955+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:16:11.003542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015863+0000","last_change":"2024-03-28T18:35:49.931394+0000","last_active":"2024-03-28T18:36:12.015863+0000","last_peered":"2024-03-28T18:36:12.015863+0000","last_clean":"2024-03-28T18:36:12.015863+0000","last_became_active":"2024-03-28T18:35:49.931045+0000","last_became_peered":"2024-03-28T18:35:49.931045+0000","last_unstale":"2024-03-28T18:36:12.015863+0000","last_undegraded":"2024-03-28T18:36:12.015863+0000","last_fullsized":"2024-03-28T18:36:12.015863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:54.030481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430381+0000","last_change":"2024-03-28T18:35:50.714745+0000","last_active":"2024-03-28T18:35:57.430381+0000","last_peered":"2024-03-28T18:35:57.430381+0000","last_clean":"2024-03-28T18:35:57.430381+0000","last_became_active":"2024-03-28T18:35:50.712991+0000","last_became_peered":"2024-03-28T18:35:50.712991+0000","last_unstale":"2024-03-28T18:35:57.430381+0000","last_undegraded":"2024-03-28T18:35:57.430381+0000","last_fullsized":"2024-03-28T18:35:57.430381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:34.403662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023963+0000","last_change":"2024-03-28T18:35:55.032353+0000","last_active":"2024-03-28T18:35:56.023963+0000","last_peered":"2024-03-28T18:35:56.023963+0000","last_clean":"2024-03-28T18:35:56.023963+0000","last_became_active":"2024-03-28T18:35:55.031889+0000","last_became_peered":"2024-03-28T18:35:55.031889+0000","last_unstale":"2024-03-28T18:35:56.023963+0000","last_undegraded":"2024-03-28T18:35:56.023963+0000","last_fullsized":"2024-03-28T18:35:56.023963+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:56.358941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022597+0000","last_change":"2024-03-28T18:35:49.944107+0000","last_active":"2024-03-28T18:35:56.022597+0000","last_peered":"2024-03-28T18:35:56.022597+0000","last_clean":"2024-03-28T18:35:56.022597+0000","last_became_active":"2024-03-28T18:35:49.943726+0000","last_became_peered":"2024-03-28T18:35:49.943726+0000","last_unstale":"2024-03-28T18:35:56.022597+0000","last_undegraded":"2024-03-28T18:35:56.022597+0000","last_fullsized":"2024-03-28T18:35:56.022597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:05.301302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379995+0000","last_change":"2024-03-28T18:35:50.709236+0000","last_active":"2024-03-28T18:35:56.379995+0000","last_peered":"2024-03-28T18:35:56.379995+0000","last_clean":"2024-03-28T18:35:56.379995+0000","last_became_active":"2024-03-28T18:35:50.708824+0000","last_became_peered":"2024-03-28T18:35:50.708824+0000","last_unstale":"2024-03-28T18:35:56.379995+0000","last_undegraded":"2024-03-28T18:35:56.379995+0000","last_fullsized":"2024-03-28T18:35:56.379995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:21:30.207033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.024213+0000","last_change":"2024-03-28T18:35:55.099711+0000","last_active":"2024-03-28T18:35:56.024213+0000","last_peered":"2024-03-28T18:35:56.024213+0000","last_clean":"2024-03-28T18:35:56.024213+0000","last_became_active":"2024-03-28T18:35:55.099054+0000","last_became_peered":"2024-03-28T18:35:55.099054+0000","last_unstale":"2024-03-28T18:35:56.024213+0000","last_undegraded":"2024-03-28T18:35:56.024213+0000","last_fullsized":"2024-03-28T18:35:56.024213+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:22:32.326655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:09.082594+0000","last_change":"2024-03-28T18:35:49.939428+0000","last_active":"2024-03-28T18:36:09.082594+0000","last_peered":"2024-03-28T18:36:09.082594+0000","last_clean":"2024-03-28T18:36:09.082594+0000","last_became_active":"2024-03-28T18:35:49.939089+0000","last_became_peered":"2024-03-28T18:35:49.939089+0000","last_unstale":"2024-03-28T18:36:09.082594+0000","last_undegraded":"2024-03-28T18:36:09.082594+0000","last_fullsized":"2024-03-28T18:36:09.082594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:38.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431212+0000","last_change":"2024-03-28T18:35:50.720931+0000","last_active":"2024-03-28T18:35:57.431212+0000","last_peered":"2024-03-28T18:35:57.431212+0000","last_clean":"2024-03-28T18:35:57.431212+0000","last_became_active":"2024-03-28T18:35:50.720544+0000","last_became_peered":"2024-03-28T18:35:50.720544+0000","last_unstale":"2024-03-28T18:35:57.431212+0000","last_undegraded":"2024-03-28T18:35:57.431212+0000","last_fullsized":"2024-03-28T18:35:57.431212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:05.092688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431122+0000","last_change":"2024-03-28T18:35:55.024518+0000","last_active":"2024-03-28T18:35:57.431122+0000","last_peered":"2024-03-28T18:35:57.431122+0000","last_clean":"2024-03-28T18:35:57.431122+0000","last_became_active":"2024-03-28T18:35:55.024103+0000","last_became_peered":"2024-03-28T18:35:55.024103+0000","last_unstale":"2024-03-28T18:35:57.431122+0000","last_undegraded":"2024-03-28T18:35:57.431122+0000","last_fullsized":"2024-03-28T18:35:57.431122+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:32:57.620744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023097+0000","last_change":"2024-03-28T18:35:49.943828+0000","last_active":"2024-03-28T18:35:56.023097+0000","last_peered":"2024-03-28T18:35:56.023097+0000","last_clean":"2024-03-28T18:35:56.023097+0000","last_became_active":"2024-03-28T18:35:49.941729+0000","last_became_peered":"2024-03-28T18:35:49.941729+0000","last_unstale":"2024-03-28T18:35:56.023097+0000","last_undegraded":"2024-03-28T18:35:56.023097+0000","last_fullsized":"2024-03-28T18:35:56.023097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:22.989447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431953+0000","last_change":"2024-03-28T18:35:50.720449+0000","last_active":"2024-03-28T18:35:57.431953+0000","last_peered":"2024-03-28T18:35:57.431953+0000","last_clean":"2024-03-28T18:35:57.431953+0000","last_became_active":"2024-03-28T18:35:50.719840+0000","last_became_peered":"2024-03-28T18:35:50.719840+0000","last_unstale":"2024-03-28T18:35:57.431953+0000","last_undegraded":"2024-03-28T18:35:57.431953+0000","last_fullsized":"2024-03-28T18:35:57.431953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:48.980677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430486+0000","last_change":"2024-03-28T18:35:55.032614+0000","last_active":"2024-03-28T18:35:57.430486+0000","last_peered":"2024-03-28T18:35:57.430486+0000","last_clean":"2024-03-28T18:35:57.430486+0000","last_became_active":"2024-03-28T18:35:55.032180+0000","last_became_peered":"2024-03-28T18:35:55.032180+0000","last_unstale":"2024-03-28T18:35:57.430486+0000","last_undegraded":"2024-03-28T18:35:57.430486+0000","last_fullsized":"2024-03-28T18:35:57.430486+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:40.121016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431318+0000","last_change":"2024-03-28T18:35:49.940681+0000","last_active":"2024-03-28T18:35:57.431318+0000","last_peered":"2024-03-28T18:35:57.431318+0000","last_clean":"2024-03-28T18:35:57.431318+0000","last_became_active":"2024-03-28T18:35:49.940376+0000","last_became_peered":"2024-03-28T18:35:49.940376+0000","last_unstale":"2024-03-28T18:35:57.431318+0000","last_undegraded":"2024-03-28T18:35:57.431318+0000","last_fullsized":"2024-03-28T18:35:57.431318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:02:11.926285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431414+0000","last_change":"2024-03-28T18:35:50.715051+0000","last_active":"2024-03-28T18:35:57.431414+0000","last_peered":"2024-03-28T18:35:57.431414+0000","last_clean":"2024-03-28T18:35:57.431414+0000","last_became_active":"2024-03-28T18:35:50.714647+0000","last_became_peered":"2024-03-28T18:35:50.714647+0000","last_unstale":"2024-03-28T18:35:57.431414+0000","last_undegraded":"2024-03-28T18:35:57.431414+0000","last_fullsized":"2024-03-28T18:35:57.431414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:02.832665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431313+0000","last_change":"2024-03-28T18:35:55.021869+0000","last_active":"2024-03-28T18:35:57.431313+0000","last_peered":"2024-03-28T18:35:57.431313+0000","last_clean":"2024-03-28T18:35:57.431313+0000","last_became_active":"2024-03-28T18:35:55.021535+0000","last_became_peered":"2024-03-28T18:35:55.021535+0000","last_unstale":"2024-03-28T18:35:57.431313+0000","last_undegraded":"2024-03-28T18:35:57.431313+0000","last_fullsized":"2024-03-28T18:35:57.431313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:31.610882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:36:12.015038+0000","last_change":"2024-03-28T18:35:49.936095+0000","last_active":"2024-03-28T18:36:12.015038+0000","last_peered":"2024-03-28T18:36:12.015038+0000","last_clean":"2024-03-28T18:36:12.015038+0000","last_became_active":"2024-03-28T18:35:49.935766+0000","last_became_peered":"2024-03-28T18:35:49.935766+0000","last_unstale":"2024-03-28T18:36:12.015038+0000","last_undegraded":"2024-03-28T18:36:12.015038+0000","last_fullsized":"2024-03-28T18:36:12.015038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:00:06.712979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431846+0000","last_change":"2024-03-28T18:35:50.715313+0000","last_active":"2024-03-28T18:35:57.431846+0000","last_peered":"2024-03-28T18:35:57.431846+0000","last_clean":"2024-03-28T18:35:57.431846+0000","last_became_active":"2024-03-28T18:35:50.714923+0000","last_became_peered":"2024-03-28T18:35:50.714923+0000","last_unstale":"2024-03-28T18:35:57.431846+0000","last_undegraded":"2024-03-28T18:35:57.431846+0000","last_fullsized":"2024-03-28T18:35:57.431846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:49.406551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430435+0000","last_change":"2024-03-28T18:35:50.714244+0000","last_active":"2024-03-28T18:35:57.430435+0000","last_peered":"2024-03-28T18:35:57.430435+0000","last_clean":"2024-03-28T18:35:57.430435+0000","last_became_active":"2024-03-28T18:35:50.713347+0000","last_became_peered":"2024-03-28T18:35:50.713347+0000","last_unstale":"2024-03-28T18:35:57.430435+0000","last_undegraded":"2024-03-28T18:35:57.430435+0000","last_fullsized":"2024-03-28T18:35:57.430435+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:34.360341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022920+0000","last_change":"2024-03-28T18:35:50.711132+0000","last_active":"2024-03-28T18:35:56.022920+0000","last_peered":"2024-03-28T18:35:56.022920+0000","last_clean":"2024-03-28T18:35:56.022920+0000","last_became_active":"2024-03-28T18:35:50.710500+0000","last_became_peered":"2024-03-28T18:35:50.710500+0000","last_unstale":"2024-03-28T18:35:56.022920+0000","last_undegraded":"2024-03-28T18:35:56.022920+0000","last_fullsized":"2024-03-28T18:35:56.022920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:22.060106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.022766+0000","last_change":"2024-03-28T18:35:50.712396+0000","last_active":"2024-03-28T18:35:56.022766+0000","last_peered":"2024-03-28T18:35:56.022766+0000","last_clean":"2024-03-28T18:35:56.022766+0000","last_became_active":"2024-03-28T18:35:50.711950+0000","last_became_peered":"2024-03-28T18:35:50.711950+0000","last_unstale":"2024-03-28T18:35:56.022766+0000","last_undegraded":"2024-03-28T18:35:56.022766+0000","last_fullsized":"2024-03-28T18:35:56.022766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:24.573754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430688+0000","last_change":"2024-03-28T18:35:50.718222+0000","last_active":"2024-03-28T18:35:57.430688+0000","last_peered":"2024-03-28T18:35:57.430688+0000","last_clean":"2024-03-28T18:35:57.430688+0000","last_became_active":"2024-03-28T18:35:50.717234+0000","last_became_peered":"2024-03-28T18:35:50.717234+0000","last_unstale":"2024-03-28T18:35:57.430688+0000","last_undegraded":"2024-03-28T18:35:57.430688+0000","last_fullsized":"2024-03-28T18:35:57.430688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:28:47.544280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179593+0000","last_change":"2024-03-28T18:35:50.707486+0000","last_active":"2024-03-28T18:35:56.179593+0000","last_peered":"2024-03-28T18:35:56.179593+0000","last_clean":"2024-03-28T18:35:56.179593+0000","last_became_active":"2024-03-28T18:35:50.706902+0000","last_became_peered":"2024-03-28T18:35:50.706902+0000","last_unstale":"2024-03-28T18:35:56.179593+0000","last_undegraded":"2024-03-28T18:35:56.179593+0000","last_fullsized":"2024-03-28T18:35:56.179593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:00.160873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430769+0000","last_change":"2024-03-28T18:35:50.715566+0000","last_active":"2024-03-28T18:35:57.430769+0000","last_peered":"2024-03-28T18:35:57.430769+0000","last_clean":"2024-03-28T18:35:57.430769+0000","last_became_active":"2024-03-28T18:35:50.715193+0000","last_became_peered":"2024-03-28T18:35:50.715193+0000","last_unstale":"2024-03-28T18:35:57.430769+0000","last_undegraded":"2024-03-28T18:35:57.430769+0000","last_fullsized":"2024-03-28T18:35:57.430769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:57.126250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.023032+0000","last_change":"2024-03-28T18:35:50.709904+0000","last_active":"2024-03-28T18:35:56.023032+0000","last_peered":"2024-03-28T18:35:56.023032+0000","last_clean":"2024-03-28T18:35:56.023032+0000","last_became_active":"2024-03-28T18:35:50.709575+0000","last_became_peered":"2024-03-28T18:35:50.709575+0000","last_unstale":"2024-03-28T18:35:56.023032+0000","last_undegraded":"2024-03-28T18:35:56.023032+0000","last_fullsized":"2024-03-28T18:35:56.023032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:26.580718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.380260+0000","last_change":"2024-03-28T18:35:50.705804+0000","last_active":"2024-03-28T18:35:56.380260+0000","last_peered":"2024-03-28T18:35:56.380260+0000","last_clean":"2024-03-28T18:35:56.380260+0000","last_became_active":"2024-03-28T18:35:50.705482+0000","last_became_peered":"2024-03-28T18:35:50.705482+0000","last_unstale":"2024-03-28T18:35:56.380260+0000","last_undegraded":"2024-03-28T18:35:56.380260+0000","last_fullsized":"2024-03-28T18:35:56.380260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:45:37.512184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430586+0000","last_change":"2024-03-28T18:35:50.707563+0000","last_active":"2024-03-28T18:35:57.430586+0000","last_peered":"2024-03-28T18:35:57.430586+0000","last_clean":"2024-03-28T18:35:57.430586+0000","last_became_active":"2024-03-28T18:35:50.706300+0000","last_became_peered":"2024-03-28T18:35:50.706300+0000","last_unstale":"2024-03-28T18:35:57.430586+0000","last_undegraded":"2024-03-28T18:35:57.430586+0000","last_fullsized":"2024-03-28T18:35:57.430586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:05.324365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.379875+0000","last_change":"2024-03-28T18:35:50.705549+0000","last_active":"2024-03-28T18:35:56.379875+0000","last_peered":"2024-03-28T18:35:56.379875+0000","last_clean":"2024-03-28T18:35:56.379875+0000","last_became_active":"2024-03-28T18:35:50.705190+0000","last_became_peered":"2024-03-28T18:35:50.705190+0000","last_unstale":"2024-03-28T18:35:56.379875+0000","last_undegraded":"2024-03-28T18:35:56.379875+0000","last_fullsized":"2024-03-28T18:35:56.379875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:38.662227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.431512+0000","last_change":"2024-03-28T18:35:50.719591+0000","last_active":"2024-03-28T18:35:57.431512+0000","last_peered":"2024-03-28T18:35:57.431512+0000","last_clean":"2024-03-28T18:35:57.431512+0000","last_became_active":"2024-03-28T18:35:50.719272+0000","last_became_peered":"2024-03-28T18:35:50.719272+0000","last_unstale":"2024-03-28T18:35:57.431512+0000","last_undegraded":"2024-03-28T18:35:57.431512+0000","last_fullsized":"2024-03-28T18:35:57.431512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:06.627761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:56.179444+0000","last_change":"2024-03-28T18:35:50.712965+0000","last_active":"2024-03-28T18:35:56.179444+0000","last_peered":"2024-03-28T18:35:56.179444+0000","last_clean":"2024-03-28T18:35:56.179444+0000","last_became_active":"2024-03-28T18:35:50.712648+0000","last_became_peered":"2024-03-28T18:35:50.712648+0000","last_unstale":"2024-03-28T18:35:56.179444+0000","last_undegraded":"2024-03-28T18:35:56.179444+0000","last_fullsized":"2024-03-28T18:35:56.179444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:00.925664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T18:35:57.430626+0000","last_change":"2024-03-28T18:35:50.717277+0000","last_active":"2024-03-28T18:35:57.430626+0000","last_peered":"2024-03-28T18:35:57.430626+0000","last_clean":"2024-03-28T18:35:57.430626+0000","last_became_active":"2024-03-28T18:35:50.716161+0000","last_became_peered":"2024-03-28T18:35:50.716161+0000","last_unstale":"2024-03-28T18:35:57.430626+0000","last_undegraded":"2024-03-28T18:35:57.430626+0000","last_fullsized":"2024-03-28T18:35:57.430626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:42.789575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210208,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5738,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":118,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":19426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318288,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":487067,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579926,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":1082233,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487851,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005088,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":487805,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266725,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":1078017,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":487789,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":490909,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":488545,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":490943,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":487497,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":991232,"data_stored":487089,"data_compressed":14648,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724590,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":1077617,"data_compressed":50230,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:36:14.858 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T18:36:14.858 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T18:36:14.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T18:36:15.145 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:15.267 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:15 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/687922659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:15.267 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:15 smithi145 ceph-mon[32151]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:15.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:15 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/687922659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:15 smithi192 ceph-mon[36589]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:15 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/687922659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:15 smithi164 ceph-mon[36248]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:15.906 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:36:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:36:15.554+0000 7f84ef9a3700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-28T18:36:15.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:36:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:36:15.554+0000 7f84ef9a3700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-28T18:36:16.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:16 smithi192 ceph-mon[36589]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:16.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:16 smithi192 ceph-mon[36589]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-28T18:36:16.422 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:16 smithi145 ceph-mon[32151]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:16.422 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:16 smithi145 ceph-mon[32151]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-28T18:36:16.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:16 smithi164 ceph-mon[36248]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:36:16.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:16 smithi164 ceph-mon[36248]: pgmap v302: 129 pgs: 129 active+clean; 587 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 -- 172.21.15.145:0/834232359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81010e0 msgr2=0x7f3ea8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 --2- 172.21.15.145:0/834232359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81010e0 0x7f3ea8101500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e980099c0 tx=0x7f3e98009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 -- 172.21.15.145:0/834232359 shutdown_connections 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 --2- 172.21.15.145:0/834232359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea8109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 --2- 172.21.15.145:0/834232359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81010e0 0x7f3ea8101500 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 --2- 172.21.15.145:0/834232359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81022e0 0x7f3ea8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 -- 172.21.15.145:0/834232359 >> 172.21.15.145:0/834232359 conn(0x7f3ea80fc680 msgr2=0x7f3ea80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 -- 172.21.15.145:0/834232359 shutdown_connections 2024-03-28T18:36:16.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.981+0000 7f3eac80c700 1 -- 172.21.15.145:0/834232359 wait complete. 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 Processor -- start 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 -- start start 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81010e0 0x7f3ea806a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea81a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea810ccd0 con 0x7f3ea81022e0 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea810cb50 con 0x7f3ea81010e0 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3eac80c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea810ce50 con 0x7f3ea8102c80 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:16.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58146/0 (socket says 172.21.15.145:58146) 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 -- 172.21.15.145:0/3976419040 learned_addr learned my addr 172.21.15.145:0/3976419040 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea6d9d700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea81a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea659c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81010e0 0x7f3ea806a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 -- 172.21.15.145:0/3976419040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 msgr2=0x7f3ea81a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea81a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 -- 172.21.15.145:0/3976419040 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81010e0 msgr2=0x7f3ea806a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81010e0 0x7f3ea806a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea5d9b700 1 -- 172.21.15.145:0/3976419040 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea81a2260 con 0x7f3ea81022e0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.982+0000 7f3ea6d9d700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea81a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3ea5d9b700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00e9a0 tx=0x7f3e9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c00ca10 con 0x7f3ea81022e0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea81a2550 con 0x7f3ea81022e0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e9c00cbb0 con 0x7f3ea81022e0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c018680 con 0x7f3ea81022e0 2024-03-28T18:36:16.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.983+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea81a2ae0 con 0x7f3ea81022e0 2024-03-28T18:36:16.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.984+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea80506a0 con 0x7f3ea81022e0 2024-03-28T18:36:16.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.985+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3e9c010430 con 0x7f3ea81022e0 2024-03-28T18:36:16.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.985+0000 7f3e977fe700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 0x7f3e900a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:16.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.986+0000 7f3ea659c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 0x7f3e900a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:16.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.986+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3e9c0cefe0 con 0x7f3ea81022e0 2024-03-28T18:36:16.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.986+0000 7f3ea659c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 0x7f3e900a4f10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3e98003790 tx=0x7f3e98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:16.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:16.990+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e9c09bce0 con 0x7f3ea81022e0 2024-03-28T18:36:17.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.274+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f3ea8062da0 con 0x7f3ea81022e0 2024-03-28T18:36:17.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.276+0000 7f3e977fe700 1 -- 172.21.15.145:0/3976419040 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3e9c09b5e0 con 0x7f3ea81022e0 2024-03-28T18:36:17.276 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-03-28T18:36:17.276 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQChuAVm4ZNqEBAAAMPJrSjkGHXquX2bmYjW0g== 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 msgr2=0x7f3e900a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 0x7f3e900a4f10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3e98003790 tx=0x7f3e98005fb0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 msgr2=0x7f3ea819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00e9a0 tx=0x7f3e9c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 shutdown_connections 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3e900a2a50 0x7f3e900a4f10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ea8102c80 0x7f3ea81a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea81010e0 0x7f3ea806a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.279 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 --2- 172.21.15.145:0/3976419040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ea81022e0 0x7f3ea819f530 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:17.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 >> 172.21.15.145:0/3976419040 conn(0x7f3ea80fc680 msgr2=0x7f3ea8105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:17.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 shutdown_connections 2024-03-28T18:36:17.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:17.279+0000 7f3eac80c700 1 -- 172.21.15.145:0/3976419040 wait complete. 2024-03-28T18:36:17.587 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:17 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:36:17.599 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:17 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:36:17.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:17 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:36:17.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:17 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:36:17.711 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:17 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:36:17.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:17 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3976419040' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:36:18.108 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:36:18.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T18:36:18.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T18:36:18.155 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T18:36:18.155 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T18:36:18.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T18:36:18.447 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:18.583 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:18 smithi145 ceph-mon[32151]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.7 KiB/s wr, 20 op/s 2024-03-28T18:36:18.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:18 smithi192 ceph-mon[36589]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.7 KiB/s wr, 20 op/s 2024-03-28T18:36:18.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:18 smithi164 ceph-mon[36248]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.7 KiB/s wr, 20 op/s 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.270+0000 7fc1d11b6700 1 -- 172.21.15.145:0/2254694534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 msgr2=0x7fc1cc10eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/2254694534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc10eff0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400b0a0 tx=0x7fc1c400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 -- 172.21.15.145:0/2254694534 shutdown_connections 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/2254694534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc10eff0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/2254694534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 0x7fc1cc108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/2254694534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 -- 172.21.15.145:0/2254694534 >> 172.21.15.145:0/2254694534 conn(0x7fc1cc079260 msgr2=0x7fc1cc079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:20.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 -- 172.21.15.145:0/2254694534 shutdown_connections 2024-03-28T18:36:20.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.271+0000 7fc1d11b6700 1 -- 172.21.15.145:0/2254694534 wait complete. 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 Processor -- start 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 -- start start 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 0x7fc1cc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:20.273 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1124f0 con 0x7fc1cc102c00 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc112370 con 0x7fc1cc102240 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1d11b6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc112670 con 0x7fc1cc1082b0 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1ca59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1cad9d700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1ca59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc19eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58166/0 (socket says 172.21.15.145:58166) 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1ca59c700 1 -- 172.21.15.145:0/554903503 learned_addr learned my addr 172.21.15.145:0/554903503 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.272+0000 7fc1cb59e700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 0x7fc1cc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 -- 172.21.15.145:0/554903503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 msgr2=0x7fc1cc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 0x7fc1cc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 -- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 msgr2=0x7fc1cc19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 -- 172.21.15.145:0/554903503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc198f90 con 0x7fc1cc102240 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1cad9d700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc19e5e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9f0 tx=0x7fc1bc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:20.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc004220 con 0x7fc1cc102240 2024-03-28T18:36:20.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc199220 con 0x7fc1cc102240 2024-03-28T18:36:20.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc004d10 con 0x7fc1cc102240 2024-03-28T18:36:20.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc005020 con 0x7fc1cc102240 2024-03-28T18:36:20.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.273+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0743c0 con 0x7fc1cc102240 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.274+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc1038d0 con 0x7fc1cc102240 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.275+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc1bc004750 con 0x7fc1cc102240 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.275+0000 7fc1b3fff700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 0x7fc1b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.276+0000 7fc1ca59c700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 0x7fc1b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.276+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc1bc0ced40 con 0x7fc1cc102240 2024-03-28T18:36:20.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.276+0000 7fc1ca59c700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 0x7fc1b40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc199e70 tx=0x7fc1c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:20.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.280+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1bc09b9e0 con 0x7fc1cc102240 2024-03-28T18:36:20.542 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.542+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc1cc02da30 con 0x7fc1cc102240 2024-03-28T18:36:20.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.546+0000 7fc1b3fff700 1 -- 172.21.15.145:0/554903503 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262663 (secure 0 0 0) 0x7fc1bc09b2e0 con 0x7fc1cc102240 2024-03-28T18:36:20.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 msgr2=0x7fc1b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 0x7fc1b40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc199e70 tx=0x7fc1c0009260 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 msgr2=0x7fc1cc19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc19e5e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00e9f0 tx=0x7fc1bc00ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 shutdown_connections 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1b40a2920 0x7fc1b40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1cc1082b0 0x7fc1cc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1cc102240 0x7fc1cc19e5e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 --2- 172.21.15.145:0/554903503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1cc102c00 0x7fc1cc19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 >> 172.21.15.145:0/554903503 conn(0x7fc1cc079260 msgr2=0x7fc1cc100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 shutdown_connections 2024-03-28T18:36:20.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:20.550+0000 7fc1d11b6700 1 -- 172.21.15.145:0/554903503 wait complete. 2024-03-28T18:36:20.873 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:20 smithi145 ceph-mon[32151]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T18:36:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:20 smithi192 ceph-mon[36589]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T18:36:20.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:20 smithi164 ceph-mon[36248]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T18:36:21.241 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":82375532},{"type":"v1","addr":"172.21.15.145:6801","nonce":82375532}]},"active_addr":"172.21.15.145:6801/82375532","active_change":"2024-03-28T18:28:53.394169+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.145:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":4218641686}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":3894963170}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":3310240071}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":3747630696}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":271047772}]}]} 2024-03-28T18:36:21.246 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T18:36:21.247 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T18:36:21.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:36:21.495 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:21.853 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:21 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/554903503' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:36:21.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:21 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/554903503' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:36:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:21 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/554903503' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:36:22.797 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:22 smithi145 ceph-mon[32151]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T18:36:22.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:22 smithi192 ceph-mon[36589]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T18:36:22.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:22 smithi164 ceph-mon[36248]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-28T18:36:23.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 -- 172.21.15.145:0/2987411676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 msgr2=0x7f2c0810df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:23.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/2987411676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0810df70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c00009f90 tx=0x7f2c0000b330 comp rx=0 tx=0).stop 2024-03-28T18:36:23.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 -- 172.21.15.145:0/2987411676 shutdown_connections 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/2987411676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0810df70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/2987411676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c081060d0 0x7f2c081064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/2987411676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c08101be0 0x7f2c08102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 -- 172.21.15.145:0/2987411676 >> 172.21.15.145:0/2987411676 conn(0x7f2c080763f0 msgr2=0x7f2c08076800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 -- 172.21.15.145:0/2987411676 shutdown_connections 2024-03-28T18:36:23.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.390+0000 7f2c0cac7700 1 -- 172.21.15.145:0/2987411676 wait complete. 2024-03-28T18:36:23.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 Processor -- start 2024-03-28T18:36:23.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 -- start start 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 0x7f2c081986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c081113d0 con 0x7f2c08101be0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08111250 con 0x7f2c081060d0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0cac7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08111550 con 0x7f2c081025a0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:47734/0 (socket says 172.21.15.145:47734) 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c0659c700 1 -- 172.21.15.145:0/1420615557 learned_addr learned my addr 172.21.15.145:0/1420615557 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c05d9b700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.391+0000 7f2c06d9d700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 0x7f2c081986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 -- 172.21.15.145:0/1420615557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 msgr2=0x7f2c081986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 0x7f2c081986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 -- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 msgr2=0x7f2c0819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 -- 172.21.15.145:0/1420615557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08198fc0 con 0x7f2c081025a0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c06d9d700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 0x7f2c081986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c0659c700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c05d9b700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0819eb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00e9f0 tx=0x7f2bfc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc00c9c0 con 0x7f2c081025a0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bfc00cb60 con 0x7f2c081025a0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc018680 con 0x7f2c081025a0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c081992b0 con 0x7f2c081025a0 2024-03-28T18:36:23.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.392+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c081a5590 con 0x7f2c081025a0 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.393+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0806a4e0 con 0x7f2c081025a0 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.394+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f2bfc047a90 con 0x7f2c081025a0 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.395+0000 7f2bf77fe700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 0x7f2bf00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.395+0000 7f2c0659c700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 0x7f2bf00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.395+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2bfc0cee70 con 0x7f2c081025a0 2024-03-28T18:36:23.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.395+0000 7f2c0659c700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 0x7f2bf00a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009750 tx=0x7f2bf8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:23.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.398+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2bfc09bb70 con 0x7f2c081025a0 2024-03-28T18:36:23.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.610+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c081a5760 con 0x7f2c081025a0 2024-03-28T18:36:23.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.612+0000 7f2bf77fe700 1 -- 172.21.15.145:0/1420615557 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7f2bfc09b470 con 0x7f2c081025a0 2024-03-28T18:36:23.612 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:23.612 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 msgr2=0x7f2bf00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 0x7f2bf00a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009750 tx=0x7f2bf8009310 comp rx=0 tx=0).stop 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 msgr2=0x7f2c0819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0819eb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00e9f0 tx=0x7f2bfc00ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 shutdown_connections 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2bf00a29a0 0x7f2bf00a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c081060d0 0x7f2c081986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c08101be0 0x7f2c0819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 --2- 172.21.15.145:0/1420615557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c081025a0 0x7f2c0819eb50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:23.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 >> 172.21.15.145:0/1420615557 conn(0x7f2c080763f0 msgr2=0x7f2c08100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:23.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 shutdown_connections 2024-03-28T18:36:23.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:23.614+0000 7f2c0cac7700 1 -- 172.21.15.145:0/1420615557 wait complete. 2024-03-28T18:36:24.529 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T18:36:24.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:36:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:24 smithi145 ceph-mon[32151]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-03-28T18:36:24.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:24 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1420615557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:24.777 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:24.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:24 smithi192 ceph-mon[36589]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-03-28T18:36:24.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:24 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1420615557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:24 smithi164 ceph-mon[36248]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-03-28T18:36:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:24 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1420615557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:26.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 -- 172.21.15.145:0/1531784781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe730107510 msgr2=0x7fe73010f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 --2- 172.21.15.145:0/1531784781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe730107510 0x7fe73010f090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe72800b0d0 tx=0x7fe72800b3e0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 -- 172.21.15.145:0/1531784781 shutdown_connections 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 --2- 172.21.15.145:0/1531784781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe730107510 0x7fe73010f090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 --2- 172.21.15.145:0/1531784781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006a930 0x7fe73006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 --2- 172.21.15.145:0/1531784781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe73006b2e0 0x7fe730106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 -- 172.21.15.145:0/1531784781 >> 172.21.15.145:0/1531784781 conn(0x7fe7300770e0 msgr2=0x7fe7300774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 -- 172.21.15.145:0/1531784781 shutdown_connections 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.515+0000 7fe738614700 1 -- 172.21.15.145:0/1531784781 wait complete. 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 Processor -- start 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 -- start start 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:26.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 0x7fe7301016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe730107510 0x7fe730101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7301124b0 con 0x7fe73006a930 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730112330 con 0x7fe730107510 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe738614700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730112630 con 0x7fe73006b2e0 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe7363b0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe735baf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 0x7fe7301016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe735baf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 0x7fe7301016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:35500/0 (socket says 172.21.15.145:35500) 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe735baf700 1 -- 172.21.15.145:0/3584783436 learned_addr learned my addr 172.21.15.145:0/3584783436 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe7363b0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:47760/0 (socket says 172.21.15.145:47760) 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.516+0000 7fe736bb1700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe730107510 0x7fe730101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 -- 172.21.15.145:0/3584783436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 msgr2=0x7fe7301016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 0x7fe7301016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 -- 172.21.15.145:0/3584783436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe730107510 msgr2=0x7fe730101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:26.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe730107510 0x7fe730101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.519 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 -- 172.21.15.145:0/3584783436 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7301ab290 con 0x7fe73006a930 2024-03-28T18:36:26.519 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7363b0700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00e9f0 tx=0x7fe72c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00cbd0 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7301ab520 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.517+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe72c00cd70 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.518+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c018680 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.518+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7301aba00 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.519+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7301038c0 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.520+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe72c010430 con 0x7fe73006a930 2024-03-28T18:36:26.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.520+0000 7fe7277fe700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 0x7fe71c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:26.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.520+0000 7fe735baf700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 0x7fe71c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:26.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.520+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe72c0cf040 con 0x7fe73006a930 2024-03-28T18:36:26.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.521+0000 7fe735baf700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 0x7fe71c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe720005e90 tx=0x7fe720005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:26.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.525+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe72c09bd40 con 0x7fe73006a930 2024-03-28T18:36:26.728 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.728+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe73002da30 con 0x7fe73006a930 2024-03-28T18:36:26.729 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.729+0000 7fe7277fe700 1 -- 172.21.15.145:0/3584783436 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24174 (secure 0 0 0) 0x7fe72c097070 con 0x7fe73006a930 2024-03-28T18:36:26.730 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:36:26.730 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":91,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:35:55.370670+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 msgr2=0x7fe71c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 0x7fe71c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe720005e90 tx=0x7fe720005d00 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 msgr2=0x7fe730101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00e9f0 tx=0x7fe72c00ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 shutdown_connections 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe71c0a2920 0x7fe71c0a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe730107510 0x7fe730101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe73006a930 0x7fe730101190 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 --2- 172.21.15.145:0/3584783436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe73006b2e0 0x7fe7301016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 >> 172.21.15.145:0/3584783436 conn(0x7fe7300770e0 msgr2=0x7fe7300ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 shutdown_connections 2024-03-28T18:36:26.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:26.732+0000 7fe738614700 1 -- 172.21.15.145:0/3584783436 wait complete. 2024-03-28T18:36:26.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:26 smithi145 ceph-mon[32151]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 21 op/s 2024-03-28T18:36:26.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:26 smithi192 ceph-mon[36589]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 21 op/s 2024-03-28T18:36:26.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:26 smithi164 ceph-mon[36248]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.8 KiB/s wr, 21 op/s 2024-03-28T18:36:27.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T18:36:27.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T18:36:27.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T18:36:27.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T18:36:27.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T18:36:27.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T18:36:27.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T18:36:27.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T18:36:27.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T18:36:27.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T18:36:27.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T18:36:27.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T18:36:27.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:27 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3584783436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:27.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:27 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3584783436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:28.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:27 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3584783436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:36:28.253 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:28.255 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:28.681 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:28 smithi145 ceph-mon[32151]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-28T18:36:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:28 smithi192 ceph-mon[36589]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-28T18:36:28.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:28 smithi164 ceph-mon[36248]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-28T18:36:29.439 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.879 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.880 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.884 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.887 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.891 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.895 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:29.895 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:30.706 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:30 smithi145 ceph-mon[32151]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-28T18:36:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:30 smithi192 ceph-mon[36589]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-28T18:36:30.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:30 smithi164 ceph-mon[36248]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-28T18:36:32.834 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:32 smithi145 ceph-mon[32151]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:36:32.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:32 smithi192 ceph-mon[36589]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:36:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:32 smithi164 ceph-mon[36248]: pgmap v310: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:36:34.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:34 smithi192 ceph-mon[36589]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:36:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:34 smithi164 ceph-mon[36248]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:36:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:34 smithi145 ceph-mon[32151]: pgmap v311: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:36:35.951 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:36.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:36 smithi145 ceph-mon[32151]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:36 smithi192 ceph-mon[36589]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:36 smithi164 ceph-mon[36248]: pgmap v312: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:38.665 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:38 smithi145 ceph-mon[32151]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:38 smithi192 ceph-mon[36589]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:38 smithi164 ceph-mon[36248]: pgmap v313: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:39.024 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 -- 172.21.15.145:0/1079660618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b0069e40 msgr2=0x7ff5b006a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/1079660618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b0069e40 0x7ff5b006a2a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00099c0 tx=0x7ff5a0009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 -- 172.21.15.145:0/1079660618 shutdown_connections 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/1079660618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b006a7e0 0x7ff5b010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/1079660618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b01081e0 0x7ff5b01085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/1079660618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b0069e40 0x7ff5b006a2a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.863+0000 7ff5b7f29700 1 -- 172.21.15.145:0/1079660618 >> 172.21.15.145:0/1079660618 conn(0x7ff5b0077fc0 msgr2=0x7ff5b00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:39.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 -- 172.21.15.145:0/1079660618 shutdown_connections 2024-03-28T18:36:39.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 -- 172.21.15.145:0/1079660618 wait complete. 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 Processor -- start 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 -- start start 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 0x7ff5b0101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b006a7e0 0x7ff5b01016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.864+0000 7ff5b7f29700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b01081e0 0x7ff5b0101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b7f29700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0112490 con 0x7ff5b006a7e0 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b7f29700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0112310 con 0x7ff5b0069e40 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b7f29700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0112610 con 0x7ff5b01081e0 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 0x7ff5b0101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:39.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 0x7ff5b0101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:55160/0 (socket says 172.21.15.145:55160) 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 -- 172.21.15.145:0/3990803575 learned_addr learned my addr 172.21.15.145:0/3990803575 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b54c4700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b006a7e0 0x7ff5b01016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b64c6700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b01081e0 0x7ff5b0101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 -- 172.21.15.145:0/3990803575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b01081e0 msgr2=0x7ff5b0101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b01081e0 0x7ff5b0101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 -- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b006a7e0 msgr2=0x7ff5b01016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b006a7e0 0x7ff5b01016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 -- 172.21.15.145:0/3990803575 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b01a6e90 con 0x7ff5b0069e40 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b54c4700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5b006a7e0 0x7ff5b01016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b64c6700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5b01081e0 0x7ff5b0101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b5cc5700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 0x7ff5b0101160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00e9a0 tx=0x7ff5ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00c9c0 con 0x7ff5b0069e40 2024-03-28T18:36:39.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5ac00cb60 con 0x7ff5b0069e40 2024-03-28T18:36:39.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b01a7180 con 0x7ff5b0069e40 2024-03-28T18:36:39.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.865+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac018680 con 0x7ff5b0069e40 2024-03-28T18:36:39.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.866+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b01a7690 con 0x7ff5b0069e40 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.868+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff5ac00cd10 con 0x7ff5b0069e40 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5a6ffd700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff59c0a2920 0x7ff59c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5b54c4700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff59c0a2920 0x7ff59c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff5ac0ceef0 con 0x7ff5b0069e40 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] conn(0x7ff5b0062ee0 0x7ff5b002c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 --> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5b010a700 con 0x7ff5b0062ee0 2024-03-28T18:36:39.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.869+0000 7ff5b54c4700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff59c0a2920 0x7ff59c0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0102bf0 tx=0x7ff5a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:39.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.870+0000 7ff5b64c6700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] conn(0x7ff5b0062ee0 0x7ff5b002c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:39.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.870+0000 7ff5b64c6700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] conn(0x7ff5b0062ee0 0x7ff5b002c8b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:39.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.871+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== osd.4 v2:172.21.15.164:6808/2404350787 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff5b010a700 con 0x7ff5b0062ee0 2024-03-28T18:36:39.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.889+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 --> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5b0103890 con 0x7ff5b0062ee0 2024-03-28T18:36:39.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5a6ffd700 1 -- 172.21.15.145:0/3990803575 <== osd.4 v2:172.21.15.164:6808/2404350787 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff5b0103890 con 0x7ff5b0062ee0 2024-03-28T18:36:39.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] conn(0x7ff5b0062ee0 msgr2=0x7ff5b002c8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] conn(0x7ff5b0062ee0 0x7ff5b002c8b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff59c0a2920 msgr2=0x7ff59c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff59c0a2920 0x7ff59c0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0102bf0 tx=0x7ff5a0005fb0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 msgr2=0x7ff5b0101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 --2- 172.21.15.145:0/3990803575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0069e40 0x7ff5b0101160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00e9a0 tx=0x7ff5ac00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b64c6700 1 -- 172.21.15.145:0/3990803575 reap_dead start 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 shutdown_connections 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 >> 172.21.15.145:0/3990803575 conn(0x7ff5b0077fc0 msgr2=0x7ff5b00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 shutdown_connections 2024-03-28T18:36:39.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.890+0000 7ff5b7f29700 1 -- 172.21.15.145:0/3990803575 wait complete. 2024-03-28T18:36:40.098 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 -- 172.21.15.145:0/258236893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 msgr2=0x7f17581075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.098 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 --2- 172.21.15.145:0/258236893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f17581075e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1748009990 tx=0x7f1748009ca0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.098 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 -- 172.21.15.145:0/258236893 shutdown_connections 2024-03-28T18:36:40.098 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 --2- 172.21.15.145:0/258236893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 0x7f175810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 --2- 172.21.15.145:0/258236893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f17581075e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 --2- 172.21.15.145:0/258236893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 0x7f1758100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 -- 172.21.15.145:0/258236893 >> 172.21.15.145:0/258236893 conn(0x7f1758077690 msgr2=0x7f17580feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 -- 172.21.15.145:0/258236893 shutdown_connections 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.070+0000 7f175c8d0700 1 -- 172.21.15.145:0/258236893 wait complete. 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 Processor -- start 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 -- start start 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 0x7f175819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 0x7f175819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f1758198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17581125e0 con 0x7f1758107200 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1758112460 con 0x7f1758101140 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175c8d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1758112760 con 0x7f1758100780 2024-03-28T18:36:40.099 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f1756d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f1758198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f1756d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f1758198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54632/0 (socket says 172.21.15.145:54632) 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f1756d9d700 1 -- 172.21.15.145:0/438682355 learned_addr learned my addr 172.21.15.145:0/438682355 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f175659c700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 0x7f175819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.071+0000 7f1755d9b700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 0x7f175819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 -- 172.21.15.145:0/438682355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 msgr2=0x7f175819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 0x7f175819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 -- 172.21.15.145:0/438682355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 msgr2=0x7f175819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.100 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 0x7f175819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758198fe0 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f175659c700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1758100780 0x7f175819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1755d9b700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1758101140 0x7f175819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f1756d9d700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f1758198880 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f175000ef90 tx=0x7f175000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17500092e0 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1758199270 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1750004bb0 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750004280 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.072+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17581a5590 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.073+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17580628a0 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.074+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f175001b030 con 0x7f1758107200 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f17477fe700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17400a2920 0x7f17400a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f175659c700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17400a2920 0x7f17400a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:40.101 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f17500cf390 con 0x7f1758107200 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f17477fe700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] conn(0x7f17400ad5e0 0x7f17400afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f175001fe50 con 0x7f17400ad5e0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17500cf740 con 0x7f1758107200 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f175659c700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17400a2920 0x7f17400a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f174800b2e0 tx=0x7f1748005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.075+0000 7f1755d9b700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] conn(0x7f17400ad5e0 0x7f17400afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.076+0000 7f1755d9b700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] conn(0x7f17400ad5e0 0x7f17400afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.076+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== osd.9 v2:172.21.15.145:6826/2302293093 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f175001fe50 con 0x7f17400ad5e0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.093+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 --> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1758062ca0 con 0x7f17400ad5e0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f17477fe700 1 -- 172.21.15.145:0/438682355 <== osd.9 v2:172.21.15.145:6826/2302293093 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1758062ca0 con 0x7f17400ad5e0 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] conn(0x7f17400ad5e0 msgr2=0x7f17400afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] conn(0x7f17400ad5e0 0x7f17400afa20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17400a2920 msgr2=0x7f17400a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17400a2920 0x7f17400a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f174800b2e0 tx=0x7f1748005960 comp rx=0 tx=0).stop 2024-03-28T18:36:40.102 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 msgr2=0x7f1758198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 --2- 172.21.15.145:0/438682355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1758107200 0x7f1758198880 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f175000ef90 tx=0x7f175000c3a0 comp rx=0 tx=0).stop 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f1756d9d700 1 -- 172.21.15.145:0/438682355 reap_dead start 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 shutdown_connections 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 >> 172.21.15.145:0/438682355 conn(0x7f1758077690 msgr2=0x7f1758105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 shutdown_connections 2024-03-28T18:36:40.103 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:40.094+0000 7f175c8d0700 1 -- 172.21.15.145:0/438682355 wait complete. 2024-03-28T18:36:40.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:40 smithi145 ceph-mon[32151]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:40.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:40 smithi192 ceph-mon[36589]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:40 smithi164 ceph-mon[36248]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:41.134 INFO:teuthology.orchestra.run.smithi145.stdout:141733920827 2024-03-28T18:36:41.134 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 -- 172.21.15.145:0/3846205656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406a6d0 msgr2=0x7f230406aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 --2- 172.21.15.145:0/3846205656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406a6d0 0x7f230406aab0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098f0 tx=0x7f22f4009c00 comp rx=0 tx=0).stop 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 -- 172.21.15.145:0/3846205656 shutdown_connections 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 --2- 172.21.15.145:0/3846205656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230410ec70 0x7f2304111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 --2- 172.21.15.145:0/3846205656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406a6d0 0x7f230406aab0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 --2- 172.21.15.145:0/3846205656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f230406aff0 0x7f230410e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.772+0000 7f2309352700 1 -- 172.21.15.145:0/3846205656 >> 172.21.15.145:0/3846205656 conn(0x7f23040770e0 msgr2=0x7f23040774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.773+0000 7f2309352700 1 -- 172.21.15.145:0/3846205656 shutdown_connections 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.773+0000 7f2309352700 1 -- 172.21.15.145:0/3846205656 wait complete. 2024-03-28T18:36:42.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.773+0000 7f2309352700 1 Processor -- start 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 -- start start 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230406a6d0 0x7f23041a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406aff0 0x7f23041a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f230410ec70 0x7f230419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2304114660 con 0x7f230406a6d0 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041144e0 con 0x7f230406aff0 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2309352700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23041147e0 con 0x7f230410ec70 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230406a6d0 0x7f23041a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2302ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230406a6d0 0x7f23041a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54598/0 (socket says 172.21.15.145:54598) 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406aff0 0x7f23041a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f2302ffd700 1 -- 172.21.15.145:0/2441279606 learned_addr learned my addr 172.21.15.145:0/2441279606 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23037fe700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f230410ec70 0x7f230419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.120 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 -- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f230410ec70 msgr2=0x7f230419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f230410ec70 0x7f230419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 -- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230406a6d0 msgr2=0x7f23041a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f230406a6d0 0x7f23041a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.774+0000 7f23027fc700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230419b120 con 0x7f230406aff0 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f23027fc700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406aff0 0x7f23041a0d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9f0 tx=0x7f22f800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8004220 con 0x7f230406aff0 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230419b410 con 0x7f230406aff0 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22f8004d10 con 0x7f230406aff0 2024-03-28T18:36:42.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8005020 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.775+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23041a76d0 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.776+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23040628a0 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.777+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f22f8004750 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.777+0000 7f22ebfff700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f22ec0a2920 0x7f22ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.777+0000 7f2302ffd700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f22ec0a2920 0x7f22ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f22f80cee50 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f22ebfff700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] conn(0x7f22ec0ad5e0 0x7f22ec0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22f800ae50 con 0x7f22ec0ad5e0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f80cf2a0 con 0x7f230406aff0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f2302ffd700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f22ec0a2920 0x7f22ec0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009680 tx=0x7f22f4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.778+0000 7f23037fe700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] conn(0x7f22ec0ad5e0 0x7f22ec0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.779+0000 7f23037fe700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] conn(0x7f22ec0ad5e0 0x7f22ec0afa20 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.780+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== osd.11 v2:172.21.15.192:6824/3207430647 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22f800ae50 con 0x7f22ec0ad5e0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.799+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 --> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2304062d20 con 0x7f22ec0ad5e0 2024-03-28T18:36:42.122 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f22ebfff700 1 -- 172.21.15.145:0/2441279606 <== osd.11 v2:172.21.15.192:6824/3207430647 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2304062d20 con 0x7f22ec0ad5e0 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] conn(0x7f22ec0ad5e0 msgr2=0x7f22ec0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] conn(0x7f22ec0ad5e0 0x7f22ec0afa20 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f22ec0a2920 msgr2=0x7f22ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f22ec0a2920 0x7f22ec0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009680 tx=0x7f22f4019040 comp rx=0 tx=0).stop 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406aff0 msgr2=0x7f23041a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 --2- 172.21.15.145:0/2441279606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230406aff0 0x7f23041a0d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9f0 tx=0x7f22f800ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f23037fe700 1 -- 172.21.15.145:0/2441279606 reap_dead start 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 shutdown_connections 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 >> 172.21.15.145:0/2441279606 conn(0x7f23040770e0 msgr2=0x7f2304110100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 shutdown_connections 2024-03-28T18:36:42.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:39.800+0000 7f2309352700 1 -- 172.21.15.145:0/2441279606 wait complete. 2024-03-28T18:36:42.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 -- 172.21.15.145:0/163896479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c102d80 msgr2=0x7f8b4c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 --2- 172.21.15.145:0/163896479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c102d80 0x7f8b4c106dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b400099c0 tx=0x7f8b40009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 -- 172.21.15.145:0/163896479 shutdown_connections 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 --2- 172.21.15.145:0/163896479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 0x7f8b4c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 --2- 172.21.15.145:0/163896479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 --2- 172.21.15.145:0/163896479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c102d80 0x7f8b4c106dd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 -- 172.21.15.145:0/163896479 >> 172.21.15.145:0/163896479 conn(0x7f8b4c0fdc40 msgr2=0x7f8b4c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 -- 172.21.15.145:0/163896479 shutdown_connections 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.699+0000 7f8b512a7700 1 -- 172.21.15.145:0/163896479 wait complete. 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 Processor -- start 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 -- start start 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 0x7f8b4c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c1124c0 con 0x7f8b4c1023d0 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c112340 con 0x7f8b4c102d80 2024-03-28T18:36:42.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b512a7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c112640 con 0x7f8b4c1074b0 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b4affd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b4a7fc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b4affd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54660/0 (socket says 172.21.15.145:54660) 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.700+0000 7f8b4affd700 1 -- 172.21.15.145:0/45051209 learned_addr learned my addr 172.21.15.145:0/45051209 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4b7fe700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 0x7f8b4c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 -- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 msgr2=0x7f8b4c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 0x7f8b4c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 -- 172.21.15.145:0/45051209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c102d80 msgr2=0x7f8b4c19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c198fc0 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4a7fc700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b4c102d80 0x7f8b4c19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4b7fe700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8b4c1074b0 0x7f8b4c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b4affd700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c19e610 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00e9a0 tx=0x7f8b3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00c9c0 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b3c00cb60 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c1992b0 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c018660 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.701+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c1a5470 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.702+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b4c062920 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.704+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8b3c00cd10 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.704+0000 7f8b33fff700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b340a2920 0x7f8b340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.704+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8b3c0cf440 con 0x7f8b4c1023d0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.704+0000 7f8b4a7fc700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b340a2920 0x7f8b340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b33fff700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] conn(0x7f8b340ad5e0 0x7f8b340afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:42.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b3c007e50 con 0x7f8b340ad5e0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b3c0cf890 con 0x7f8b4c1023d0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b4b7fe700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] conn(0x7f8b340ad5e0 0x7f8b340afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b4a7fc700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b340a2920 0x7f8b340a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.705+0000 7f8b4b7fe700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] conn(0x7f8b340ad5e0 0x7f8b340afa20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.707+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== osd.5 v2:172.21.15.192:6808/22255675 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8b3c007e50 con 0x7f8b340ad5e0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.723+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 --> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b4c062d50 con 0x7f8b340ad5e0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.723+0000 7f8b33fff700 1 -- 172.21.15.145:0/45051209 <== osd.5 v2:172.21.15.192:6808/22255675 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8b4c062d50 con 0x7f8b340ad5e0 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] conn(0x7f8b340ad5e0 msgr2=0x7f8b340afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] conn(0x7f8b340ad5e0 0x7f8b340afa20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b340a2920 msgr2=0x7f8b340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8b340a2920 0x7f8b340a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b40005fb0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 msgr2=0x7f8b4c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 --2- 172.21.15.145:0/45051209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c1023d0 0x7f8b4c19e610 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00e9a0 tx=0x7f8b3c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b4b7fe700 1 -- 172.21.15.145:0/45051209 reap_dead start 2024-03-28T18:36:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 shutdown_connections 2024-03-28T18:36:42.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 >> 172.21.15.145:0/45051209 conn(0x7f8b4c0fdc40 msgr2=0x7f8b4c0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:42.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 shutdown_connections 2024-03-28T18:36:42.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:41.724+0000 7f8b512a7700 1 -- 172.21.15.145:0/45051209 wait complete. 2024-03-28T18:36:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:42 smithi145 ceph-mon[32151]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:42.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:42 smithi192 ceph-mon[36589]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:42.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:42 smithi164 ceph-mon[36248]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:36:44.143 INFO:teuthology.orchestra.run.smithi145.stdout:352187318295 2024-03-28T18:36:44.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T18:36:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:44 smithi145 ceph-mon[32151]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T18:36:44.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:44 smithi192 ceph-mon[36589]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T18:36:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:44 smithi164 ceph-mon[36248]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- 172.21.15.145:0/1940394222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06b480 msgr2=0x7f173c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 --2- 172.21.15.145:0/1940394222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06b480 0x7f173c106df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f172c009990 tx=0x7f172c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- 172.21.15.145:0/1940394222 shutdown_connections 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 --2- 172.21.15.145:0/1940394222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c1074d0 0x7f173c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 --2- 172.21.15.145:0/1940394222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f173c06aad0 0x7f173c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 --2- 172.21.15.145:0/1940394222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06b480 0x7f173c106df0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- 172.21.15.145:0/1940394222 >> 172.21.15.145:0/1940394222 conn(0x7f173c0770e0 msgr2=0x7f173c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- 172.21.15.145:0/1940394222 shutdown_connections 2024-03-28T18:36:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- 172.21.15.145:0/1940394222 wait complete. 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 Processor -- start 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.104+0000 7f1744091700 1 -- start start 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 0x7f173c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c06b480 0x7f173c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f173c1074d0 0x7f173c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c112450 con 0x7f173c06aad0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c1122d0 con 0x7f173c06b480 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c1125d0 con 0x7f173c1074d0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 0x7f173c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f174162c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c06b480 0x7f173c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 0x7f173c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53388/0 (socket says 172.21.15.145:53388) 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 -- 172.21.15.145:0/2073633838 learned_addr learned my addr 172.21.15.145:0/2073633838 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f174262e700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f173c1074d0 0x7f173c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 -- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f173c1074d0 msgr2=0x7f173c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f173c1074d0 0x7f173c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 -- 172.21.15.145:0/2073633838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c06b480 msgr2=0x7f173c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c06b480 0x7f173c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c198f50 con 0x7f173c06aad0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f174162c700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f173c06b480 0x7f173c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1741e2d700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 0x7f173c19e5a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f173800e9a0 tx=0x7f173800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:46.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173800ca10 con 0x7f173c06aad0 2024-03-28T18:36:46.109 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.105+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c199240 con 0x7f173c06aad0 2024-03-28T18:36:46.109 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.106+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f173800cbb0 con 0x7f173c06aad0 2024-03-28T18:36:46.109 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.106+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738018680 con 0x7f173c06aad0 2024-03-28T18:36:46.109 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.106+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c1a5520 con 0x7f173c06aad0 2024-03-28T18:36:46.109 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.106+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f173c0ffac0 con 0x7f173c06aad0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1738010430 con 0x7f173c06aad0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f1732ffd700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17280a2920 0x7f17280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f17380cf880 con 0x7f173c06aad0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f1732ffd700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] conn(0x7f17280ad5e0 0x7f17280afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f173801bec0 con 0x7f17280ad5e0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.109+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17380cfcd0 con 0x7f173c06aad0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.108+0000 7f174162c700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17280a2920 0x7f17280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.109+0000 7f174262e700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] conn(0x7f17280ad5e0 0x7f17280afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.109+0000 7f174262e700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] conn(0x7f17280ad5e0 0x7f17280afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:46.111 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.110+0000 7f174162c700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17280a2920 0x7f17280a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f172c006010 tx=0x7f172c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:46.111 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.110+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== osd.2 v2:172.21.15.192:6800/629223859 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f173801bec0 con 0x7f17280ad5e0 2024-03-28T18:36:46.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.127+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 --> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f173c06a0a0 con 0x7f17280ad5e0 2024-03-28T18:36:46.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.127+0000 7f1732ffd700 1 -- 172.21.15.145:0/2073633838 <== osd.2 v2:172.21.15.192:6800/629223859 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f173c06a0a0 con 0x7f17280ad5e0 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] conn(0x7f17280ad5e0 msgr2=0x7f17280afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] conn(0x7f17280ad5e0 0x7f17280afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17280a2920 msgr2=0x7f17280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17280a2920 0x7f17280a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f172c006010 tx=0x7f172c005960 comp rx=0 tx=0).stop 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 msgr2=0x7f173c19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 --2- 172.21.15.145:0/2073633838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f173c06aad0 0x7f173c19e5a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f173800e9a0 tx=0x7f173800ecb0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f174262e700 1 -- 172.21.15.145:0/2073633838 reap_dead start 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 shutdown_connections 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 >> 172.21.15.145:0/2073633838 conn(0x7f173c0770e0 msgr2=0x7f173c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 shutdown_connections 2024-03-28T18:36:46.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.128+0000 7f1744091700 1 -- 172.21.15.145:0/2073633838 wait complete. 2024-03-28T18:36:46.814 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:46 smithi145 ceph-mon[32151]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-28T18:36:46.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:46 smithi192 ceph-mon[36589]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-28T18:36:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:46 smithi164 ceph-mon[36248]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 -- 172.21.15.145:0/1403316687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006a930 msgr2=0x7f97f006ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1403316687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006a930 0x7f97f006ad10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009990 tx=0x7f97d8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 -- 172.21.15.145:0/1403316687 shutdown_connections 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1403316687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 0x7f97f010f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1403316687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006a930 0x7f97f006ad10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1403316687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006b2e0 0x7f97f0106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 -- 172.21.15.145:0/1403316687 >> 172.21.15.145:0/1403316687 conn(0x7f97f00770e0 msgr2=0x7f97f00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 -- 172.21.15.145:0/1403316687 shutdown_connections 2024-03-28T18:36:47.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 -- 172.21.15.145:0/1403316687 wait complete. 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.473+0000 7f97f6a38700 1 Processor -- start 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 -- start start 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006a930 0x7f97f0101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 0x7f97f01016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 0x7f97f0101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f01124c0 con 0x7f97f006b2e0 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0112340 con 0x7f97f006a930 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f6a38700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0112640 con 0x7f97f01074a0 2024-03-28T18:36:47.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97ef7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 0x7f97f01016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97ef7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 0x7f97f01016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53398/0 (socket says 172.21.15.145:53398) 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97ef7fe700 1 -- 172.21.15.145:0/1193741512 learned_addr learned my addr 172.21.15.145:0/1193741512 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97f4fd5700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 0x7f97f0101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.474+0000 7f97effff700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006a930 0x7f97f0101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 -- 172.21.15.145:0/1193741512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006a930 msgr2=0x7f97f0101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006a930 0x7f97f0101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 -- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 msgr2=0x7f97f01016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 0x7f97f01016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f01a6e10 con 0x7f97f01074a0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97effff700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f006a930 0x7f97f0101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97ef7fe700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f006b2e0 0x7f97f01016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f4fd5700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 0x7f97f0101c10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f97e800efc0 tx=0x7f97e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e80092e0 con 0x7f97f01074a0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f01a70a0 con 0x7f97f01074a0 2024-03-28T18:36:47.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97e8004bb0 con 0x7f97f01074a0 2024-03-28T18:36:47.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8004280 con 0x7f97f01074a0 2024-03-28T18:36:47.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.475+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f01a7590 con 0x7f97f01074a0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.476+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97f00628e0 con 0x7f97f01074a0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.477+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f97e801b030 con 0x7f97f01074a0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.477+0000 7f97ed7fa700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f97dc0a2920 0x7f97dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97effff700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f97dc0a2920 0x7f97dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f97e80cea40 con 0x7f97f01074a0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ed7fa700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] conn(0x7f97dc0ad5e0 0x7f97dc0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97e8013ec0 con 0x7f97dc0ad5e0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97e80cedf0 con 0x7f97f01074a0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ef7fe700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] conn(0x7f97dc0ad5e0 0x7f97dc0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97effff700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f97dc0a2920 0x7f97dc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97d800b2e0 tx=0x7f97d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.478+0000 7f97ef7fe700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] conn(0x7f97dc0ad5e0 0x7f97dc0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.479+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== osd.3 v2:172.21.15.145:6810/1910028611 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f97e8013ec0 con 0x7f97dc0ad5e0 2024-03-28T18:36:47.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.496+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 --> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f97f0062ce0 con 0x7f97dc0ad5e0 2024-03-28T18:36:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.496+0000 7f97ed7fa700 1 -- 172.21.15.145:0/1193741512 <== osd.3 v2:172.21.15.145:6810/1910028611 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f97f0062ce0 con 0x7f97dc0ad5e0 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.496+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] conn(0x7f97dc0ad5e0 msgr2=0x7f97dc0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.496+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] conn(0x7f97dc0ad5e0 0x7f97dc0afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f97dc0a2920 msgr2=0x7f97dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f97dc0a2920 0x7f97dc0a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97d800b2e0 tx=0x7f97d8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 msgr2=0x7f97f0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 --2- 172.21.15.145:0/1193741512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f01074a0 0x7f97f0101c10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f97e800efc0 tx=0x7f97e800c3a0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f4fd5700 1 -- 172.21.15.145:0/1193741512 reap_dead start 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 shutdown_connections 2024-03-28T18:36:47.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 >> 172.21.15.145:0/1193741512 conn(0x7f97f00770e0 msgr2=0x7f97f00ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:47.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 shutdown_connections 2024-03-28T18:36:47.499 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.497+0000 7f97f6a38700 1 -- 172.21.15.145:0/1193741512 wait complete. 2024-03-28T18:36:47.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 -- 172.21.15.145:0/4204193325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 msgr2=0x7f5bb806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/4204193325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb806a2a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0099c0 tx=0x7f5bac009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 -- 172.21.15.145:0/4204193325 shutdown_connections 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/4204193325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 0x7f5bb810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/4204193325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 0x7f5bb81085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/4204193325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb806a2a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 -- 172.21.15.145:0/4204193325 >> 172.21.15.145:0/4204193325 conn(0x7f5bb8077fc0 msgr2=0x7f5bb80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 -- 172.21.15.145:0/4204193325 shutdown_connections 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.008+0000 7f5bbcf39700 1 -- 172.21.15.145:0/4204193325 wait complete. 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.009+0000 7f5bbcf39700 1 Processor -- start 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.009+0000 7f5bbcf39700 1 -- start start 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb8101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 0x7f5bb81016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 0x7f5bb8101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb8112440 con 0x7f5bb8069e40 2024-03-28T18:36:47.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb81122c0 con 0x7f5bb806a7e0 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bbcf39700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb81125c0 con 0x7f5bb81081e0 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb8101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb8101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53368/0 (socket says 172.21.15.145:53368) 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb659c700 1 -- 172.21.15.145:0/1278127508 learned_addr learned my addr 172.21.15.145:0/1278127508 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 0x7f5bb8101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb5d9b700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 0x7f5bb81016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 -- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 msgr2=0x7f5bb81016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 0x7f5bb81016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 -- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 msgr2=0x7f5bb8101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb8101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb81a6ec0 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb659c700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bb8069e40 0x7f5bb8101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb5d9b700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bb806a7e0 0x7f5bb81016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.010+0000 7f5bb6d9d700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 0x7f5bb8101be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000efb0 tx=0x7f5bb000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0003a00 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bb0003ba0 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb81a71b0 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0004850 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb81a7670 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.011+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bb80628a0 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.013+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5bb0012050 con 0x7f5bb81081e0 2024-03-28T18:36:47.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5ba77fe700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5ba00a2920 0x7f5ba00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5bb659c700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5ba00a2920 0x7f5ba00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5bb00cec50 con 0x7f5bb81081e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5ba77fe700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] conn(0x7f5ba00ad5e0 0x7f5ba00afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ba00b00f0 con 0x7f5ba00ad5e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.014+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bb00cf0a0 con 0x7f5bb81081e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.015+0000 7f5bb5d9b700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] conn(0x7f5ba00ad5e0 0x7f5ba00afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.015+0000 7f5bb659c700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5ba00a2920 0x7f5ba00a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8011550 tx=0x7f5ba8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.015+0000 7f5bb5d9b700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] conn(0x7f5ba00ad5e0 0x7f5ba00afa20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.016+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== osd.10 v2:172.21.15.164:6824/3964117002 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ba00b00f0 con 0x7f5ba00ad5e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 --> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5bb8062ca0 con 0x7f5ba00ad5e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5ba77fe700 1 -- 172.21.15.145:0/1278127508 <== osd.10 v2:172.21.15.164:6824/3964117002 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5bb8062ca0 con 0x7f5ba00ad5e0 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] conn(0x7f5ba00ad5e0 msgr2=0x7f5ba00afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] conn(0x7f5ba00ad5e0 0x7f5ba00afa20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5ba00a2920 msgr2=0x7f5ba00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5ba00a2920 0x7f5ba00a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8011550 tx=0x7f5ba8009260 comp rx=0 tx=0).stop 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 msgr2=0x7f5bb8101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.032+0000 7f5bbcf39700 1 --2- 172.21.15.145:0/1278127508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bb81081e0 0x7f5bb8101be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000efb0 tx=0x7f5bb000c3a0 comp rx=0 tx=0).stop 2024-03-28T18:36:47.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.033+0000 7f5bb6d9d700 1 -- 172.21.15.145:0/1278127508 reap_dead start 2024-03-28T18:36:47.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.033+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 shutdown_connections 2024-03-28T18:36:47.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.033+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 >> 172.21.15.145:0/1278127508 conn(0x7f5bb8077fc0 msgr2=0x7f5bb80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:47.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.033+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 shutdown_connections 2024-03-28T18:36:47.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:46.033+0000 7f5bbcf39700 1 -- 172.21.15.145:0/1278127508 wait complete. 2024-03-28T18:36:48.835 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:48 smithi145 ceph-mon[32151]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:36:48.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:48 smithi192 ceph-mon[36589]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:36:48.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:48 smithi164 ceph-mon[36248]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:36:49.773 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:49.799 INFO:teuthology.orchestra.run.smithi145.stdout:317827579934 2024-03-28T18:36:49.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T18:36:49.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:49.910 INFO:teuthology.orchestra.run.smithi145.stdout:107374182463 2024-03-28T18:36:49.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T18:36:49.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 -- 172.21.15.145:0/743696878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71981081e0 msgr2=0x7f71981085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 --2- 172.21.15.145:0/743696878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71981081e0 0x7f71981085c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f71940099c0 tx=0x7f7194009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 -- 172.21.15.145:0/743696878 shutdown_connections 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 --2- 172.21.15.145:0/743696878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f719806a7e0 0x7f719810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 --2- 172.21.15.145:0/743696878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71981081e0 0x7f71981085c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 --2- 172.21.15.145:0/743696878 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7198069e40 0x7f719806a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 -- 172.21.15.145:0/743696878 >> 172.21.15.145:0/743696878 conn(0x7f7198077fc0 msgr2=0x7f71980783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 -- 172.21.15.145:0/743696878 shutdown_connections 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 -- 172.21.15.145:0/743696878 wait complete. 2024-03-28T18:36:50.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.116+0000 7f719f3fd700 1 Processor -- start 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 -- start start 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 0x7f7198101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f719806a7e0 0x7f71981016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71981081e0 0x7f7198101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198112440 con 0x7f7198069e40 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71981122c0 con 0x7f71981081e0 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719f3fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71981125c0 con 0x7f719806a7e0 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719d199700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 0x7f7198101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719d199700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 0x7f7198101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53348/0 (socket says 172.21.15.145:53348) 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f719806a7e0 0x7f71981016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719d199700 1 -- 172.21.15.145:0/1155680798 learned_addr learned my addr 172.21.15.145:0/1155680798 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719d99a700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71981081e0 0x7f7198101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 -- 172.21.15.145:0/1155680798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71981081e0 msgr2=0x7f7198101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71981081e0 0x7f7198101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 -- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 msgr2=0x7f7198101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 0x7f7198101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71981a6e00 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719d199700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7198069e40 0x7f7198101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.117+0000 7f719c998700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f719806a7e0 0x7f71981016a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9f0 tx=0x7f718800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188004220 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71981a70f0 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7188004d10 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188005020 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71981a75e0 con 0x7f719806a7e0 2024-03-28T18:36:50.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.118+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71980628a0 con 0x7f719806a7e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.120+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7188004750 con 0x7f719806a7e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.120+0000 7f718e7fc700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f71840a2920 0x7f71840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.120+0000 7f719d199700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f71840a2920 0x7f71840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f71880cefe0 con 0x7f719806a7e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f718e7fc700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] conn(0x7f71840ad5e0 0x7f71840afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7188012ec0 con 0x7f71840ad5e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71880cf390 con 0x7f719806a7e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f719d99a700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] conn(0x7f71840ad5e0 0x7f71840afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f719d199700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f71840a2920 0x7f71840a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7194009990 tx=0x7f7194005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.121+0000 7f719d99a700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] conn(0x7f71840ad5e0 0x7f71840afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.122+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== osd.8 v2:172.21.15.192:6816/1092086458 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7188012ec0 con 0x7f71840ad5e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 --> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7198062d20 con 0x7f71840ad5e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f718e7fc700 1 -- 172.21.15.145:0/1155680798 <== osd.8 v2:172.21.15.192:6816/1092086458 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7198062d20 con 0x7f71840ad5e0 2024-03-28T18:36:50.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] conn(0x7f71840ad5e0 msgr2=0x7f71840afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] conn(0x7f71840ad5e0 0x7f71840afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f71840a2920 msgr2=0x7f71840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f71840a2920 0x7f71840a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7194009990 tx=0x7f7194005960 comp rx=0 tx=0).stop 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f719806a7e0 msgr2=0x7f71981016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 --2- 172.21.15.145:0/1155680798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f719806a7e0 0x7f71981016a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9f0 tx=0x7f718800ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719d99a700 1 -- 172.21.15.145:0/1155680798 reap_dead start 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 shutdown_connections 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 >> 172.21.15.145:0/1155680798 conn(0x7f7198077fc0 msgr2=0x7f71980ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 shutdown_connections 2024-03-28T18:36:50.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:45.140+0000 7f719f3fd700 1 -- 172.21.15.145:0/1155680798 wait complete. 2024-03-28T18:36:50.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:50 smithi192 ceph-mon[36589]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:36:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:50 smithi164 ceph-mon[36248]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:36:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.980 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:50 smithi145 ceph-mon[32151]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:36:50.980 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.980 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.981 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:50.981 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:36:51.007 INFO:teuthology.orchestra.run.smithi145.stdout:261993005095 2024-03-28T18:36:51.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T18:36:51.469 INFO:teuthology.orchestra.run.smithi145.stdout:296352743458 2024-03-28T18:36:51.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 -- 172.21.15.145:0/1549101913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0106370 msgr2=0x7f44a010df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 --2- 172.21.15.145:0/1549101913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0106370 0x7f44a010df40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4498009f90 tx=0x7f449800b330 comp rx=0 tx=0).stop 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 -- 172.21.15.145:0/1549101913 shutdown_connections 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 --2- 172.21.15.145:0/1549101913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0106370 0x7f44a010df40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 --2- 172.21.15.145:0/1549101913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101320 0x7f44a0101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 --2- 172.21.15.145:0/1549101913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44a0101cd0 0x7f44a0105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 -- 172.21.15.145:0/1549101913 >> 172.21.15.145:0/1549101913 conn(0x7f44a00770e0 msgr2=0x7f44a00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 -- 172.21.15.145:0/1549101913 shutdown_connections 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.711+0000 7f44a5937700 1 -- 172.21.15.145:0/1549101913 wait complete. 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 Processor -- start 2024-03-28T18:36:51.713 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 -- start start 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0101320 0x7f44a01996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 0x7f44a0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44a0106370 0x7f44a01a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0111360 con 0x7f44a0106370 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a01111e0 con 0x7f44a0101320 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f44a5937700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a01114e0 con 0x7f44a0101cd0 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f449e7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 0x7f44a0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:51.715 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f449f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44a0106370 0x7f44a01a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f449e7fc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 0x7f44a0199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:36436/0 (socket says 172.21.15.145:36436) 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f449e7fc700 1 -- 172.21.15.145:0/3905180475 learned_addr learned my addr 172.21.15.145:0/3905180475 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.712+0000 7f449effd700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0101320 0x7f44a01996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 -- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0101320 msgr2=0x7f44a01996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44a0101320 0x7f44a01996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 -- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44a0106370 msgr2=0x7f44a01a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44a0106370 0x7f44a01a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a01a4ca0 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f449e7fc700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 0x7f44a0199c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f449400ea30 tx=0x7f449400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449400cbd0 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4494004500 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a01a4ed0 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494003e20 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.713+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a01a53d0 con 0x7f44a0101cd0 2024-03-28T18:36:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.714+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44a0062860 con 0x7f44a0101cd0 2024-03-28T18:36:51.717 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.715+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f4494004020 con 0x7f44a0101cd0 2024-03-28T18:36:51.717 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f44a4935700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44880a29a0 0x7f44880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f44940cee50 con 0x7f44a0101cd0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f449effd700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44880a29a0 0x7f44880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f44a4935700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] conn(0x7f44880ad660 0x7f44880afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44880b0170 con 0x7f44880ad660 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44940d1510 con 0x7f44a0101cd0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f449f7fe700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] conn(0x7f44880ad660 0x7f44880afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.716+0000 7f449effd700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44880a29a0 0x7f44880a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4490005fd0 tx=0x7f4490009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.717+0000 7f449f7fe700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] conn(0x7f44880ad660 0x7f44880afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:51.718 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.717+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== osd.7 v2:172.21.15.164:6816/795549152 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f44880b0170 con 0x7f44880ad660 2024-03-28T18:36:51.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.739+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 --> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f44a0062d10 con 0x7f44880ad660 2024-03-28T18:36:51.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.739+0000 7f44a4935700 1 -- 172.21.15.145:0/3905180475 <== osd.7 v2:172.21.15.164:6816/795549152 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f44a0062d10 con 0x7f44880ad660 2024-03-28T18:36:51.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] conn(0x7f44880ad660 msgr2=0x7f44880afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] conn(0x7f44880ad660 0x7f44880afaa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:51.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44880a29a0 msgr2=0x7f44880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44880a29a0 0x7f44880a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4490005fd0 tx=0x7f4490009310 comp rx=0 tx=0).stop 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 msgr2=0x7f44a0199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 --2- 172.21.15.145:0/3905180475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a0101cd0 0x7f44a0199c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f449400ea30 tx=0x7f449400ed40 comp rx=0 tx=0).stop 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f449f7fe700 1 -- 172.21.15.145:0/3905180475 reap_dead start 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 shutdown_connections 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 >> 172.21.15.145:0/3905180475 conn(0x7f44a00770e0 msgr2=0x7f44a00ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 shutdown_connections 2024-03-28T18:36:51.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:51.740+0000 7f44a5937700 1 -- 172.21.15.145:0/3905180475 wait complete. 2024-03-28T18:36:52.534 INFO:teuthology.orchestra.run.smithi145.stdout:77309411398 2024-03-28T18:36:52.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T18:36:52.763 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:52 smithi145 ceph-mon[32151]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:36:52.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 -- 172.21.15.145:0/3131454449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 msgr2=0x7fb498106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/3131454449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb498106e00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb4880099c0 tx=0x7fb488009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 -- 172.21.15.145:0/3131454449 shutdown_connections 2024-03-28T18:36:52.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/3131454449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4981074e0 0x7fb49810f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/3131454449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 0x7fb49806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/3131454449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb498106e00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 -- 172.21.15.145:0/3131454449 >> 172.21.15.145:0/3131454449 conn(0x7fb498078160 msgr2=0x7fb498078570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 -- 172.21.15.145:0/3131454449 shutdown_connections 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.936+0000 7fb4a05e7700 1 -- 172.21.15.145:0/3131454449 wait complete. 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 Processor -- start 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 -- start start 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 0x7fb49819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb49819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4981074e0 0x7fb498198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb498112510 con 0x7fb49806b550 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb498112390 con 0x7fb49806aba0 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb4a05e7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb498112690 con 0x7fb4981074e0 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb49819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:52.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49e383700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 0x7fb49819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb49819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53408/0 (socket says 172.21.15.145:53408) 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 -- 172.21.15.145:0/1211302546 learned_addr learned my addr 172.21.15.145:0/1211302546 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49eb84700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4981074e0 0x7fb498198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 -- 172.21.15.145:0/1211302546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4981074e0 msgr2=0x7fb498198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4981074e0 0x7fb498198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 -- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 msgr2=0x7fb49819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 0x7fb49819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498198f70 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49e383700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb49806aba0 0x7fb49819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.937+0000 7fb49db82700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb49819eb50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb48800f690 tx=0x7fb488005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.938+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488027070 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.938+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb498199200 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.938+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb488003680 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.938+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4880173f0 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.938+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4981a5470 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.939+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb498062920 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.940+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb48802a070 con 0x7fb49806b550 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb48f7fe700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4840a2920 0x7fb4840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb49e383700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4840a2920 0x7fb4840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:52.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb4880d0340 con 0x7fb49806b550 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb48f7fe700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] conn(0x7fb4840ad5e0 0x7fb4840afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb488001d30 con 0x7fb4840ad5e0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4880d0790 con 0x7fb49806b550 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.941+0000 7fb49e383700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4840a2920 0x7fb4840a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb494011490 tx=0x7fb494009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.942+0000 7fb49eb84700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] conn(0x7fb4840ad5e0 0x7fb4840afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.942+0000 7fb49eb84700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] conn(0x7fb4840ad5e0 0x7fb4840afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.943+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== osd.1 v2:172.21.15.164:6800/3569377863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb488001d30 con 0x7fb4840ad5e0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 --> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb498062d20 con 0x7fb4840ad5e0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb48f7fe700 1 -- 172.21.15.145:0/1211302546 <== osd.1 v2:172.21.15.164:6800/3569377863 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb498062d20 con 0x7fb4840ad5e0 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] conn(0x7fb4840ad5e0 msgr2=0x7fb4840afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] conn(0x7fb4840ad5e0 0x7fb4840afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4840a2920 msgr2=0x7fb4840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4840a2920 0x7fb4840a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb494011490 tx=0x7fb494009260 comp rx=0 tx=0).stop 2024-03-28T18:36:52.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 msgr2=0x7fb49819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb4a05e7700 1 --2- 172.21.15.145:0/1211302546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb49806b550 0x7fb49819eb50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb48800f690 tx=0x7fb488005f10 comp rx=0 tx=0).stop 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.960+0000 7fb49eb84700 1 -- 172.21.15.145:0/1211302546 reap_dead start 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.961+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 shutdown_connections 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.961+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 >> 172.21.15.145:0/1211302546 conn(0x7fb498078160 msgr2=0x7fb4980fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.961+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 shutdown_connections 2024-03-28T18:36:52.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:47.961+0000 7fb4a05e7700 1 -- 172.21.15.145:0/1211302546 wait complete. 2024-03-28T18:36:52.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:52 smithi192 ceph-mon[36589]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:36:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:52 smithi164 ceph-mon[36248]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:36:53.493 INFO:teuthology.orchestra.run.smithi145.stdout:55834574923 2024-03-28T18:36:53.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T18:36:53.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-28T18:36:53.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-28T18:36:53.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T18:36:53.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-28T18:36:53.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T18:36:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-28T18:36:53.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-28T18:36:54.255 INFO:teuthology.orchestra.run.smithi145.stdout:227633266734 2024-03-28T18:36:54.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T18:36:54.890 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:36:54.890 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-28T18:36:54.891 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:54 smithi145 ceph-mon[32151]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 -- 172.21.15.145:0/1543087970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101090 msgr2=0x7f75181014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 --2- 172.21.15.145:0/1543087970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101090 0x7f75181014f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75080098f0 tx=0x7f7508009c00 comp rx=0 tx=0).stop 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 -- 172.21.15.145:0/1543087970 shutdown_connections 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 --2- 172.21.15.145:0/1543087970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7518101a30 0x7f751810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 --2- 172.21.15.145:0/1543087970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75181070d0 0x7f75181074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 --2- 172.21.15.145:0/1543087970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101090 0x7f75181014f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 -- 172.21.15.145:0/1543087970 >> 172.21.15.145:0/1543087970 conn(0x7f7518077fc0 msgr2=0x7f75180783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 -- 172.21.15.145:0/1543087970 shutdown_connections 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.890+0000 7f751fba7700 1 -- 172.21.15.145:0/1543087970 wait complete. 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 Processor -- start 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 -- start start 2024-03-28T18:36:54.893 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 0x7f75181997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101a30 0x7f7518199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75181070d0 0x7f75181a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518111350 con 0x7f7518101090 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75181111d0 con 0x7f75181070d0 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751fba7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75181114d0 con 0x7f7518101a30 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751d943700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 0x7f75181997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751d943700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 0x7f75181997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36746/0 (socket says 172.21.15.145:36746) 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751d943700 1 -- 172.21.15.145:0/2509668062 learned_addr learned my addr 172.21.15.145:0/2509668062 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751d142700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101a30 0x7f7518199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.891+0000 7f751e144700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75181070d0 0x7f75181a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 -- 172.21.15.145:0/2509668062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101a30 msgr2=0x7f7518199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101a30 0x7f7518199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 -- 172.21.15.145:0/2509668062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75181070d0 msgr2=0x7f75181a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75181070d0 0x7f75181a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75181a4d90 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d142700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7518101a30 0x7f7518199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751e144700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75181070d0 0x7f75181a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751d943700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 0x7f75181997d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f751400ec10 tx=0x7f751400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751400cc90 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7518197620 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f751400ce30 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514010690 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.892+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518197b80 con 0x7f7518101090 2024-03-28T18:36:54.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.893+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7518062860 con 0x7f7518101090 2024-03-28T18:36:54.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.894+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7514003e10 con 0x7f7518101090 2024-03-28T18:36:54.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f750effd700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f75040a2920 0x7f75040a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:54.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f751d142700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f75040a2920 0x7f75040a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:54.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f75140cf470 con 0x7f7518101090 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f750effd700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] conn(0x7f75040ada80 0x7f75040aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7514022e50 con 0x7f75040ada80 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.895+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75140cf8c0 con 0x7f7518101090 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.896+0000 7f751e144700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] conn(0x7f75040ada80 0x7f75040aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.896+0000 7f751d142700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f75040a2920 0x7f75040a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f75080098c0 tx=0x7f7508019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.896+0000 7f751e144700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] conn(0x7f75040ada80 0x7f75040aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:54.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.897+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== osd.0 v2:172.21.15.145:6802/2900391364 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7514022e50 con 0x7f75040ada80 2024-03-28T18:36:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:36:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-28T18:36:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:54 smithi192 ceph-mon[36589]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:36:54.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.914+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 --> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7518062ce0 con 0x7f75040ada80 2024-03-28T18:36:54.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f750effd700 1 -- 172.21.15.145:0/2509668062 <== osd.0 v2:172.21.15.145:6802/2900391364 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7518062ce0 con 0x7f75040ada80 2024-03-28T18:36:54.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] conn(0x7f75040ada80 msgr2=0x7f75040aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] conn(0x7f75040ada80 0x7f75040aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f75040a2920 msgr2=0x7f75040a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f75040a2920 0x7f75040a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f75080098c0 tx=0x7f7508019040 comp rx=0 tx=0).stop 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 msgr2=0x7f75181997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 --2- 172.21.15.145:0/2509668062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7518101090 0x7f75181997d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f751400ec10 tx=0x7f751400ef20 comp rx=0 tx=0).stop 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751e144700 1 -- 172.21.15.145:0/2509668062 reap_dead start 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 shutdown_connections 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.915+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 >> 172.21.15.145:0/2509668062 conn(0x7f7518077fc0 msgr2=0x7f75180ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.916+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 shutdown_connections 2024-03-28T18:36:54.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:54.916+0000 7f751fba7700 1 -- 172.21.15.145:0/2509668062 wait complete. 2024-03-28T18:36:54.936 INFO:teuthology.orchestra.run.smithi145.stdout:167503724597 2024-03-28T18:36:54.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 8]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-28T18:36:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-28T18:36:54.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:54 smithi164 ceph-mon[36248]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 -- 172.21.15.145:0/1445383699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103240 msgr2=0x7f0e08103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/1445383699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103240 0x7f0e08103620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009990 tx=0x7f0df8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 -- 172.21.15.145:0/1445383699 shutdown_connections 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/1445383699 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e0810c670 0x7f0e0810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/1445383699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103240 0x7f0e08103620 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/1445383699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e08103b60 0x7f0e0810c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 -- 172.21.15.145:0/1445383699 >> 172.21.15.145:0/1445383699 conn(0x7f0e080fca50 msgr2=0x7f0e080fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 -- 172.21.15.145:0/1445383699 shutdown_connections 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.139+0000 7f0e0d59d700 1 -- 172.21.15.145:0/1445383699 wait complete. 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.140+0000 7f0e0d59d700 1 Processor -- start 2024-03-28T18:36:55.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.140+0000 7f0e0d59d700 1 -- start start 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.140+0000 7f0e0d59d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 0x7f0e0819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0d59d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103b60 0x7f0e0819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0d59d700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e0810c670 0x7f0e08198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0d59d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08112400 con 0x7f0e08103b60 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0d59d700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08112280 con 0x7f0e0810c670 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0d59d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08112580 con 0x7f0e08103240 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e07fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 0x7f0e0819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e07fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 0x7f0e0819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52086/0 (socket says 172.21.15.145:52086) 2024-03-28T18:36:55.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e07fff700 1 -- 172.21.15.145:0/2291651276 learned_addr learned my addr 172.21.15.145:0/2291651276 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e077fe700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103b60 0x7f0e0819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e0810c670 0x7f0e08198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 -- 172.21.15.145:0/2291651276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 msgr2=0x7f0e0819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 0x7f0e0819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 -- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103b60 msgr2=0x7f0e0819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e08103b60 0x7f0e0819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08198fe0 con 0x7f0e0810c670 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e07fff700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e08103240 0x7f0e0819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.141+0000 7f0e0cd9c700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e0810c670 0x7f0e08198700 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000efb0 tx=0x7f0e0000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.142+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00003a00 con 0x7f0e0810c670 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.142+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e00003ba0 con 0x7f0e0810c670 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.142+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00004850 con 0x7f0e0810c670 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.142+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e08199270 con 0x7f0e0810c670 2024-03-28T18:36:55.143 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.142+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e08100dd0 con 0x7f0e0810c670 2024-03-28T18:36:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.143+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e08062920 con 0x7f0e0810c670 2024-03-28T18:36:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.144+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0e00012050 con 0x7f0e0810c670 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e057fa700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0df00a6db0 0x7f0df00a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e07fff700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0df00a6db0 0x7f0df00a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0e000ce7a0 con 0x7f0e0810c670 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e057fa700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] conn(0x7f0df00b1f10 0x7f0df00b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0df00b4a80 con 0x7f0df00b1f10 2024-03-28T18:36:55.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e000d5050 con 0x7f0e0810c670 2024-03-28T18:36:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.145+0000 7f0e077fe700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] conn(0x7f0df00b1f10 0x7f0df00b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.146+0000 7f0e07fff700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0df00a6db0 0x7f0df00a9270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009750 tx=0x7f0df800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.146+0000 7f0e077fe700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] conn(0x7f0df00b1f10 0x7f0df00b43b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:55.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.147+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== osd.6 v2:172.21.15.145:6818/2385620152 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0df00b4a80 con 0x7f0df00b1f10 2024-03-28T18:36:55.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.168+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 --> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e08062da0 con 0x7f0df00b1f10 2024-03-28T18:36:55.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.169+0000 7f0e057fa700 1 -- 172.21.15.145:0/2291651276 <== osd.6 v2:172.21.15.145:6818/2385620152 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0e08062da0 con 0x7f0df00b1f10 2024-03-28T18:36:55.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.169+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] conn(0x7f0df00b1f10 msgr2=0x7f0df00b43b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.169+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] conn(0x7f0df00b1f10 0x7f0df00b43b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0df00a6db0 msgr2=0x7f0df00a9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0df00a6db0 0x7f0df00a9270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009750 tx=0x7f0df800b280 comp rx=0 tx=0).stop 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e0810c670 msgr2=0x7f0e08198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 --2- 172.21.15.145:0/2291651276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e0810c670 0x7f0e08198700 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000efb0 tx=0x7f0e0000c3a0 comp rx=0 tx=0).stop 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0cd9c700 1 -- 172.21.15.145:0/2291651276 reap_dead start 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 shutdown_connections 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 >> 172.21.15.145:0/2291651276 conn(0x7f0e080fca50 msgr2=0x7f0e080fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 shutdown_connections 2024-03-28T18:36:55.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:36:55.170+0000 7f0e0d59d700 1 -- 172.21.15.145:0/2291651276 wait complete. 2024-03-28T18:36:55.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:55 smithi192 ceph-mon[36589]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:36:55.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:55 smithi164 ceph-mon[36248]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:36:55.984 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:55 smithi145 ceph-mon[32151]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:36:56.009 INFO:teuthology.orchestra.run.smithi145.stdout:34359738449 2024-03-28T18:36:56.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T18:36:56.590 INFO:teuthology.orchestra.run.smithi145.stdout:201863462964 2024-03-28T18:36:56.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T18:36:56.646 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.652 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.654 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.654 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.656 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.669 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:56.839 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:56 smithi145 ceph-mon[32151]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:36:56.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:56 smithi192 ceph-mon[36589]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:36:56.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:56 smithi164 ceph-mon[36248]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:36:58.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:36:58 smithi145 ceph-mon[32151]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:36:58.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:36:58 smithi192 ceph-mon[36589]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:36:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:36:58 smithi164 ceph-mon[36248]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:00 smithi192 ceph-mon[36589]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:00.912 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:00 smithi145 ceph-mon[32151]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:00.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:00 smithi164 ceph-mon[36248]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:02.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:02 smithi192 ceph-mon[36589]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:02 smithi164 ceph-mon[36248]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:02.977 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:02 smithi145 ceph-mon[32151]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5 op/s; 160 B/s, 1 objects/s recovering 2024-03-28T18:37:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:04 smithi145 ceph-mon[32151]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 129 B/s, 1 objects/s recovering 2024-03-28T18:37:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/889264307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:37:04.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:04 smithi192 ceph-mon[36589]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 129 B/s, 1 objects/s recovering 2024-03-28T18:37:04.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/889264307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:37:04.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:04 smithi164 ceph-mon[36248]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 129 B/s, 1 objects/s recovering 2024-03-28T18:37:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/889264307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 -- 172.21.15.145:0/3289114134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101a30 msgr2=0x7fc9b410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/3289114134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101a30 0x7fc9b410df10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00b0a0 tx=0x7fc9ac00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 -- 172.21.15.145:0/3289114134 shutdown_connections 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/3289114134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101a30 0x7fc9b410df10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/3289114134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 0x7fc9b41074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/3289114134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101090 0x7fc9b41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 -- 172.21.15.145:0/3289114134 >> 172.21.15.145:0/3289114134 conn(0x7fc9b4077fc0 msgr2=0x7fc9b40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 -- 172.21.15.145:0/3289114134 shutdown_connections 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 -- 172.21.15.145:0/3289114134 wait complete. 2024-03-28T18:37:05.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.189+0000 7fc9bc77b700 1 Processor -- start 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 -- start start 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 0x7fc9b419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 0x7fc9b4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4111370 con 0x7fc9b4101090 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41111f0 con 0x7fc9b41070d0 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bc77b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41114f0 con 0x7fc9b4101a30 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9ba517700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9ba517700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53962/0 (socket says 172.21.15.145:53962) 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 0x7fc9b419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9ba517700 1 -- 172.21.15.145:0/4155136922 learned_addr learned my addr 172.21.15.145:0/4155136922 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9bad18700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 0x7fc9b4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 -- 172.21.15.145:0/4155136922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 msgr2=0x7fc9b4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 0x7fc9b4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 -- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 msgr2=0x7fc9b419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 -- 172.21.15.145:0/4155136922 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4199030 con 0x7fc9b4101a30 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9ba517700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.190+0000 7fc9b9d16700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 0x7fc9b419ebc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400e9f0 tx=0x7fc9a400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.191+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a400cbd0 con 0x7fc9b4101a30 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.191+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b4199320 con 0x7fc9b4101a30 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.191+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9a400cd70 con 0x7fc9b4101a30 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.191+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4018710 con 0x7fc9b4101a30 2024-03-28T18:37:05.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.191+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b41a5600 con 0x7fc9b4101a30 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.192+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b4102780 con 0x7fc9b4101a30 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.193+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc9a4010430 con 0x7fc9b4101a30 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.193+0000 7fc9ab7fe700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 0x7fc9a00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.193+0000 7fc9ba517700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 0x7fc9a00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.193+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc9a40cf190 con 0x7fc9b4101a30 2024-03-28T18:37:05.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.194+0000 7fc9ba517700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 0x7fc9a00a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0011550 tx=0x7fc9b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:05.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.197+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc9a409bd10 con 0x7fc9b4101a30 2024-03-28T18:37:05.416 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.416+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fc9b41a57d0 con 0x7fc9b4101a30 2024-03-28T18:37:05.416 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.416+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/4155136922 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc9a409b610 con 0x7fc9b4101a30 2024-03-28T18:37:05.416 INFO:teuthology.orchestra.run.smithi145.stdout:317827579937 2024-03-28T18:37:05.418 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.418+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 msgr2=0x7fc9a00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.418 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.418+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 0x7fc9a00a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0011550 tx=0x7fc9b0009260 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.418+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 msgr2=0x7fc9b419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.418+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 0x7fc9b419ebc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400e9f0 tx=0x7fc9a400ed00 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 shutdown_connections 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9a00a2920 0x7fc9a00a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9b41070d0 0x7fc9b4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9b4101090 0x7fc9b419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 --2- 172.21.15.145:0/4155136922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9b4101a30 0x7fc9b419ebc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 >> 172.21.15.145:0/4155136922 conn(0x7fc9b4077fc0 msgr2=0x7fc9b41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 shutdown_connections 2024-03-28T18:37:05.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:05.419+0000 7fc9bc77b700 1 -- 172.21.15.145:0/4155136922 wait complete. 2024-03-28T18:37:05.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4155136922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:37:05.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- 172.21.15.145:0/581817986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006b2e0 msgr2=0x7f18b0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 --2- 172.21.15.145:0/581817986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006b2e0 0x7f18b0106dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009990 tx=0x7f18a0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- 172.21.15.145:0/581817986 shutdown_connections 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 --2- 172.21.15.145:0/581817986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b010f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 --2- 172.21.15.145:0/581817986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006a930 0x7f18b006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 --2- 172.21.15.145:0/581817986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006b2e0 0x7f18b0106dc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- 172.21.15.145:0/581817986 >> 172.21.15.145:0/581817986 conn(0x7f18b00770e0 msgr2=0x7f18b00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- 172.21.15.145:0/581817986 shutdown_connections 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- 172.21.15.145:0/581817986 wait complete. 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 Processor -- start 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.194+0000 7f18b7811700 1 -- start start 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006b2e0 0x7f18b019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b0112440 con 0x7f18b006a930 2024-03-28T18:37:05.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01122c0 con 0x7f18b01074a0 2024-03-28T18:37:05.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b7811700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01125c0 con 0x7f18b006b2e0 2024-03-28T18:37:05.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53950/0 (socket says 172.21.15.145:53950) 2024-03-28T18:37:05.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 -- 172.21.15.145:0/889264307 learned_addr learned my addr 172.21.15.145:0/889264307 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:05.918 INFO:teuthology.orchestra.run.smithi145.stdout:296352743462 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b5dae700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b4dac700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006b2e0 0x7f18b019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 -- 172.21.15.145:0/889264307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006b2e0 msgr2=0x7f18b019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006b2e0 0x7f18b019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 -- 172.21.15.145:0/889264307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 msgr2=0x7f18b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 -- 172.21.15.145:0/889264307 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b0198fc0 con 0x7f18b006a930 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b5dae700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18b55ad700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e9c0 tx=0x7f18ac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.195+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac00cbd0 con 0x7f18b006a930 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.196+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b0199250 con 0x7f18b006a930 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.196+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b0101290 con 0x7f18b006a930 2024-03-28T18:37:05.919 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.196+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18ac00ee60 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.196+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac005630 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.197+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b01038a0 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.198+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f18ac004750 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.199+0000 7f18a67fc700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 0x7f189c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.199+0000 7f18b4dac700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 0x7f189c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.199+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f18ac0cf400 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.199+0000 7f18b4dac700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 0x7f189c0a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f18a0006010 tx=0x7f18a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.202+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f18ac053740 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.432+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f18b0199c10 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.432+0000 7f18a67fc700 1 -- 172.21.15.145:0/889264307 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f18ac09bfe0 con 0x7f18b006a930 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 msgr2=0x7f189c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 0x7f189c0a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f18a0006010 tx=0x7f18a0005960 comp rx=0 tx=0).stop 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 msgr2=0x7f18b019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e9c0 tx=0x7f18ac00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 shutdown_connections 2024-03-28T18:37:05.920 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f189c0a2920 0x7f189c0a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18b01074a0 0x7f18b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b006a930 0x7f18b019e610 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 --2- 172.21.15.145:0/889264307 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18b006b2e0 0x7f18b019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 >> 172.21.15.145:0/889264307 conn(0x7f18b00770e0 msgr2=0x7f18b00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 shutdown_connections 2024-03-28T18:37:05.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:04.435+0000 7f18b7811700 1 -- 172.21.15.145:0/889264307 wait complete. 2024-03-28T18:37:05.921 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4155136922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:37:05.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4155136922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:37:06.354 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:06.354 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:06.355 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:06.356 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:06.719 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:06 smithi145 ceph-mon[32151]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5 op/s; 117 B/s, 1 objects/s recovering 2024-03-28T18:37:06.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:06 smithi192 ceph-mon[36589]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5 op/s; 117 B/s, 1 objects/s recovering 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 -- 172.21.15.145:0/1851081104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c1081e0 msgr2=0x7fa52c1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 --2- 172.21.15.145:0/1851081104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c1081e0 0x7fa52c1085c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa52400b0d0 tx=0x7fa52400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 -- 172.21.15.145:0/1851081104 shutdown_connections 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 --2- 172.21.15.145:0/1851081104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c06a7e0 0x7fa52c10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 --2- 172.21.15.145:0/1851081104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c1081e0 0x7fa52c1085c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 --2- 172.21.15.145:0/1851081104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c069e40 0x7fa52c06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 -- 172.21.15.145:0/1851081104 >> 172.21.15.145:0/1851081104 conn(0x7fa52c077fc0 msgr2=0x7fa52c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 -- 172.21.15.145:0/1851081104 shutdown_connections 2024-03-28T18:37:06.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.921+0000 7fa531e3f700 1 -- 172.21.15.145:0/1851081104 wait complete. 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 Processor -- start 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 -- start start 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 0x7fa52c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c06a7e0 0x7fa52c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c1016a0 con 0x7fa52c1081e0 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c101520 con 0x7fa52c06a7e0 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa531e3f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c101820 con 0x7fa52c069e40 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 0x7fa52c1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53972/0 (socket says 172.21.15.145:53972) 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52bfff700 1 -- 172.21.15.145:0/3571218013 learned_addr learned my addr 172.21.15.145:0/3571218013 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:06.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52affd700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c06a7e0 0x7fa52c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 -- 172.21.15.145:0/3571218013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c06a7e0 msgr2=0x7fa52c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c06a7e0 0x7fa52c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 -- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 msgr2=0x7fa52c19afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52b7fe700 1 -- 172.21.15.145:0/3571218013 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c19b770 con 0x7fa52c069e40 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.922+0000 7fa52bfff700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa52b7fe700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 0x7fa52c1a0860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa524000f80 tx=0x7fa524007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524003c00 con 0x7fa52c069e40 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa524022070 con 0x7fa52c069e40 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c19ba00 con 0x7fa52c069e40 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52401b6c0 con 0x7fa52c069e40 2024-03-28T18:37:06.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.923+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c1a76c0 con 0x7fa52c069e40 2024-03-28T18:37:06.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.924+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c103890 con 0x7fa52c069e40 2024-03-28T18:37:06.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.925+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa52404eb70 con 0x7fa52c069e40 2024-03-28T18:37:06.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.925+0000 7fa528ff9700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 0x7fa5180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:06.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.926+0000 7fa52affd700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 0x7fa5180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:06.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.926+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa52400f020 con 0x7fa52c069e40 2024-03-28T18:37:06.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.926+0000 7fa52affd700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 0x7fa5180a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa52c106df0 tx=0x7fa514009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:06.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:06.929+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa52409c870 con 0x7fa52c069e40 2024-03-28T18:37:06.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:06 smithi164 ceph-mon[36248]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 5 op/s; 117 B/s, 1 objects/s recovering 2024-03-28T18:37:07.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.149+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa52c1a7890 con 0x7fa52c069e40 2024-03-28T18:37:07.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.149+0000 7fa528ff9700 1 -- 172.21.15.145:0/3571218013 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa52409c170 con 0x7fa52c069e40 2024-03-28T18:37:07.149 INFO:teuthology.orchestra.run.smithi145.stdout:55834574927 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 msgr2=0x7fa5180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 0x7fa5180a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa52c106df0 tx=0x7fa514009260 comp rx=0 tx=0).stop 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 msgr2=0x7fa52c1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 0x7fa52c1a0860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa524000f80 tx=0x7fa524007a20 comp rx=0 tx=0).stop 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 shutdown_connections 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa5180a29a0 0x7fa5180a4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c1081e0 0x7fa52c19afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:07.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa52c069e40 0x7fa52c1a0860 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:07.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 --2- 172.21.15.145:0/3571218013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa52c06a7e0 0x7fa52c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:07.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 >> 172.21.15.145:0/3571218013 conn(0x7fa52c077fc0 msgr2=0x7fa52c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:07.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 shutdown_connections 2024-03-28T18:37:07.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:07.152+0000 7fa531e3f700 1 -- 172.21.15.145:0/3571218013 wait complete. 2024-03-28T18:37:07.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:07 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3571218013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:37:07.917 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:07 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3571218013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:37:07.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574927 for osd.1 2024-03-28T18:37:07.943 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:07.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:07 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3571218013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:37:08.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:08 smithi145 ceph-mon[32151]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:08.842 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743458 got 296352743462 for osd.9 2024-03-28T18:37:08.842 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:08.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:08 smithi192 ceph-mon[36589]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:08.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:08 smithi164 ceph-mon[36248]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:09.669 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1963511847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:37:09.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:09 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1963511847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:37:09.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:09 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1963511847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3152195580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 msgr2=0x7fc9ac105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3152195580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac105cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009990 tx=0x7fc9a0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3152195580 shutdown_connections 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3152195580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3152195580 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3152195580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac105cd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3152195580 >> 172.21.15.145:0/3152195580 conn(0x7fc9ac0770b0 msgr2=0x7fc9ac0774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3152195580 shutdown_connections 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.504+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3152195580 wait complete. 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 Processor -- start 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 -- start start 2024-03-28T18:37:10.591 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac1a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac111370 con 0x7fc9ac101320 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.505+0000 7fc9b21dc700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac1111f0 con 0x7fc9ac101cd0 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9b21dc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac1114f0 con 0x7fc9ac1063b0 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54004/0 (socket says 172.21.15.145:54004) 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/3823370816 learned_addr learned my addr 172.21.15.145:0/3823370816 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9aaffd700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9abfff700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac1a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/3823370816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 msgr2=0x7fc9ac1a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac1a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/3823370816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 msgr2=0x7fc9ac199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.592 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 -- 172.21.15.145:0/3823370816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac1a4d90 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9abfff700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac1a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9aaffd700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9ab7fe700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e9a0 tx=0x7fc99c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00ca10 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac1a4f60 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc99c00cbb0 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c018680 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.506+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac1a53b0 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.508+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac1027b0 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.509+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc99c010430 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.510+0000 7fc9a8ff9700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 0x7fc9940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.510+0000 7fc9aaffd700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 0x7fc9940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.510+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc99c0cf0f0 con 0x7fc9ac101320 2024-03-28T18:37:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.510+0000 7fc9aaffd700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 0x7fc9940a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0006010 tx=0x7fc9a0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:10.594 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.518+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc99c09bcd0 con 0x7fc9ac101320 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- 172.21.15.145:0/2584661002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc102ba0 msgr2=0x7ff4cc10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/2584661002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc10efe0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400b0d0 tx=0x7ff4c400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- 172.21.15.145:0/2584661002 shutdown_connections 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/2584661002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc10efe0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/2584661002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1081c0 0x7ff4cc1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.691 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/2584661002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- 172.21.15.145:0/2584661002 >> 172.21.15.145:0/2584661002 conn(0x7ff4cc0792c0 msgr2=0x7ff4cc0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- 172.21.15.145:0/2584661002 shutdown_connections 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- 172.21.15.145:0/2584661002 wait complete. 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 Processor -- start 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.168+0000 7ff4d30c8700 1 -- start start 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc1081c0 0x7ff4cc198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d0e64700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d1665700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc1081c0 0x7ff4cc198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc112460 con 0x7ff4cc1081c0 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d0e64700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52152/0 (socket says 172.21.15.145:52152) 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc1122e0 con 0x7ff4cc102ba0 2024-03-28T18:37:10.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc1125e0 con 0x7ff4cc1021e0 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d0e64700 1 -- 172.21.15.145:0/1963511847 learned_addr learned my addr 172.21.15.145:0/1963511847 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stdout:261993005100 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 -- 172.21.15.145:0/1963511847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 msgr2=0x7ff4cc19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 -- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc1081c0 msgr2=0x7ff4cc198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc1081c0 0x7ff4cc198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4cbfff700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc198f20 con 0x7ff4cc102ba0 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.169+0000 7ff4d0e64700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:10.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4cbfff700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc19eab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9f0 tx=0x7ff4b800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800cb70 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc199210 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b800ee90 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8003dc0 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.170+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc1a54f0 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.171+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc103870 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.172+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff4b8004000 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.173+0000 7ff4c9ffb700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 0x7ff4bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.173+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff4b80cf120 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.173+0000 7ff4d0e64700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 0x7ff4bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.175+0000 7ff4d0e64700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 0x7ff4bc0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011590 tx=0x7ff4c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.177+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4b809bca0 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.416+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7ff4cc199b70 con 0x7ff4cc102ba0 2024-03-28T18:37:10.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.416+0000 7ff4c9ffb700 1 -- 172.21.15.145:0/1963511847 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff4b809b5a0 con 0x7ff4cc102ba0 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.418+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 msgr2=0x7ff4bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.418+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 0x7ff4bc0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011590 tx=0x7ff4c0009260 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.418+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 msgr2=0x7ff4cc19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.418+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc19eab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9f0 tx=0x7ff4b800ed00 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 shutdown_connections 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff4bc0a2920 0x7ff4bc0a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4cc1081c0 0x7ff4cc198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4cc1021e0 0x7ff4cc19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 --2- 172.21.15.145:0/1963511847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4cc102ba0 0x7ff4cc19eab0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 >> 172.21.15.145:0/1963511847 conn(0x7ff4cc0792c0 msgr2=0x7ff4cc100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 shutdown_connections 2024-03-28T18:37:10.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:09.419+0000 7ff4d30c8700 1 -- 172.21.15.145:0/1963511847 wait complete. 2024-03-28T18:37:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:10 smithi145 ceph-mon[32151]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:10.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.801+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc9ac02db50 con 0x7fc9ac101320 2024-03-28T18:37:10.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.801+0000 7fc9a8ff9700 1 -- 172.21.15.145:0/3823370816 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc99c09b5d0 con 0x7fc9ac101320 2024-03-28T18:37:10.801 INFO:teuthology.orchestra.run.smithi145.stdout:77309411403 2024-03-28T18:37:10.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 msgr2=0x7fc9940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 0x7fc9940a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0006010 tx=0x7fc9a0005960 comp rx=0 tx=0).stop 2024-03-28T18:37:10.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 msgr2=0x7fc9ac1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:10.803 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e9a0 tx=0x7fc99c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 shutdown_connections 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9940a2920 0x7fc9940a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9ac1063b0 0x7fc9ac1a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9ac101320 0x7fc9ac1997d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 --2- 172.21.15.145:0/3823370816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9ac101cd0 0x7fc9ac199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.803+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 >> 172.21.15.145:0/3823370816 conn(0x7fc9ac0770b0 msgr2=0x7fc9ac107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.804+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 shutdown_connections 2024-03-28T18:37:10.804 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.804+0000 7fc9b21dc700 1 -- 172.21.15.145:0/3823370816 wait complete. 2024-03-28T18:37:10.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:10 smithi192 ceph-mon[36589]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:10.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:10 smithi164 ceph-mon[36248]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-28T18:37:11.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3823370816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:37:11.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1697458646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:37:11.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005095 got 261993005100 for osd.8 2024-03-28T18:37:11.829 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:11.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3823370816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:37:11.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1697458646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- 172.21.15.145:0/2228296633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90081070d0 msgr2=0x7f90081074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 --2- 172.21.15.145:0/2228296633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90081070d0 0x7f90081074b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009990 tx=0x7f8ff8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- 172.21.15.145:0/2228296633 shutdown_connections 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 --2- 172.21.15.145:0/2228296633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9008101a30 0x7f900810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 --2- 172.21.15.145:0/2228296633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90081070d0 0x7f90081074b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 --2- 172.21.15.145:0/2228296633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101090 0x7f90081014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- 172.21.15.145:0/2228296633 >> 172.21.15.145:0/2228296633 conn(0x7f9008077fc0 msgr2=0x7f90080783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- 172.21.15.145:0/2228296633 shutdown_connections 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- 172.21.15.145:0/2228296633 wait complete. 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 Processor -- start 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.599+0000 7f900e695700 1 -- start start 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 0x7f9008196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 0x7f9008196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081102e0 con 0x7f90081070d0 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9008110160 con 0x7f9008101090 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900e695700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9008110460 con 0x7f9008101a30 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f90077fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 0x7f9008196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54026/0 (socket says 172.21.15.145:54026) 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 -- 172.21.15.145:0/1697458646 learned_addr learned my addr 172.21.15.145:0/1697458646 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:11.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f9007fff700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 0x7f9008196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 -- 172.21.15.145:0/1697458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 msgr2=0x7f9008196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 0x7f9008196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 -- 172.21.15.145:0/1697458646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 msgr2=0x7f9008196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 0x7f9008196750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f900cc32700 1 -- 172.21.15.145:0/1697458646 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900819b760 con 0x7f90081070d0 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f9007fff700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 0x7f9008196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.600+0000 7f90077fe700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 0x7f9008196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f900cc32700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90000103c0 tx=0x7f90000106d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stdout:167503724603 2024-03-28T18:37:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000010dc0 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900819b9f0 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9000004bb0 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000009970 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.601+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90081a32d0 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.602+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4005330 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.604+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f900001d030 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.605+0000 7f90057fa700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 0x7f8ff00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.605+0000 7f9007fff700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 0x7f8ff00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.605+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f90000d1310 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.605+0000 7f9007fff700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 0x7f8ff00a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b2e0 tx=0x7f8ff8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.611+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9000024020 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.832+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8ff40058f0 con 0x7f90081070d0 2024-03-28T18:37:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.832+0000 7f90057fa700 1 -- 172.21.15.145:0/1697458646 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9000012040 con 0x7f90081070d0 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 msgr2=0x7f8ff00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 0x7f8ff00a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b2e0 tx=0x7f8ff8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 msgr2=0x7f900819b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f90000103c0 tx=0x7f90000106d0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 shutdown_connections 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8ff00a29a0 0x7f8ff00a4e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90081070d0 0x7f900819b020 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9008101090 0x7f9008196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 --2- 172.21.15.145:0/1697458646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008101a30 0x7f9008196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 >> 172.21.15.145:0/1697458646 conn(0x7f9008077fc0 msgr2=0x7f90080fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 shutdown_connections 2024-03-28T18:37:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:10.835+0000 7f900e695700 1 -- 172.21.15.145:0/1697458646 wait complete. 2024-03-28T18:37:11.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3823370816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:37:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1697458646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:37:12.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724597 got 167503724603 for osd.5 2024-03-28T18:37:12.525 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:12.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:12 smithi145 ceph-mon[32151]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:12.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:12 smithi192 ceph-mon[36589]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:12.956 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579934 got 317827579937 for osd.10 2024-03-28T18:37:12.956 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:12.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:12 smithi164 ceph-mon[36248]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:14.414 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:14.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 -- 172.21.15.145:0/3023928656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 msgr2=0x7f9f8806ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:14.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/3023928656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f8806ad10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009990 tx=0x7f9f84009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 -- 172.21.15.145:0/3023928656 shutdown_connections 2024-03-28T18:37:14.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/3023928656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 0x7f9f8810f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/3023928656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f8806ad10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/3023928656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 -- 172.21.15.145:0/3023928656 >> 172.21.15.145:0/3023928656 conn(0x7f9f880770e0 msgr2=0x7f9f880774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:14.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 -- 172.21.15.145:0/3023928656 shutdown_connections 2024-03-28T18:37:14.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.613+0000 7f9f8f97b700 1 -- 172.21.15.145:0/3023928656 wait complete. 2024-03-28T18:37:14.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.614+0000 7f9f8f97b700 1 Processor -- start 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.614+0000 7f9f8f97b700 1 -- start start 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.614+0000 7f9f8f97b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8f97b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8f97b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 0x7f9f881017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8f97b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f88112470 con 0x7f9f8806a930 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8f97b700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881122f0 con 0x7f9f8806b2e0 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8f97b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881125f0 con 0x7f9f881074a0 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37362/0 (socket says 172.21.15.145:37362) 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 -- 172.21.15.145:0/2458563478 learned_addr learned my addr 172.21.15.145:0/2458563478 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8cf16700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8df18700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 0x7f9f881017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 -- 172.21.15.145:0/2458563478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 msgr2=0x7f9f881017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 0x7f9f881017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.616 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 -- 172.21.15.145:0/2458563478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 msgr2=0x7f9f88101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 -- 172.21.15.145:0/2458563478 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f881ab270 con 0x7f9f8806a930 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8cf16700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f8d717700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f8400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.615+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84027070 con 0x7f9f8806a930 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.616+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f881ab4a0 con 0x7f9f8806a930 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.616+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f84003d70 con 0x7f9f8806a930 2024-03-28T18:37:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.616+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f881ab8f0 con 0x7f9f8806a930 2024-03-28T18:37:14.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.616+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400f460 con 0x7f9f8806a930 2024-03-28T18:37:14.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.617+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f881038a0 con 0x7f9f8806a930 2024-03-28T18:37:14.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.618+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9f8402a070 con 0x7f9f8806a930 2024-03-28T18:37:14.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.619+0000 7f9f7e7fc700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 0x7f9f740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:14.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.619+0000 7f9f8cf16700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 0x7f9f740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:14.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.619+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9f840d0030 con 0x7f9f8806a930 2024-03-28T18:37:14.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.619+0000 7f9f8cf16700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 0x7f9f740a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102900 tx=0x7f9f78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:14.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.623+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f84013070 con 0x7f9f8806a930 2024-03-28T18:37:14.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.856+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9f8802dbf0 con 0x7f9f8806a930 2024-03-28T18:37:14.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.856+0000 7f9f7e7fc700 1 -- 172.21.15.145:0/2458563478 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9f8409cca0 con 0x7f9f8806a930 2024-03-28T18:37:14.857 INFO:teuthology.orchestra.run.smithi145.stdout:227633266738 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.858+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 msgr2=0x7f9f740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.858+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 0x7f9f740a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9f88102900 tx=0x7f9f78009260 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.858+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 msgr2=0x7f9f88102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.858+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f84000c00 tx=0x7f9f8400bab0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 shutdown_connections 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9f740a29a0 0x7f9f740a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f881074a0 0x7f9f881017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f8806a930 0x7f9f88102d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 --2- 172.21.15.145:0/2458563478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f8806b2e0 0x7f9f88101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 >> 172.21.15.145:0/2458563478 conn(0x7f9f880770e0 msgr2=0x7f9f880ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 shutdown_connections 2024-03-28T18:37:14.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:14.859+0000 7f9f8f97b700 1 -- 172.21.15.145:0/2458563478 wait complete. 2024-03-28T18:37:14.894 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:14 smithi145 ceph-mon[32151]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:14.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:14 smithi192 ceph-mon[36589]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:14 smithi164 ceph-mon[36248]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:15.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:15 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2458563478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:37:15.909 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:15 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2458563478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:37:15.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:15 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2458563478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:37:16.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:16 smithi192 ceph-mon[36589]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:16.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:16 smithi164 ceph-mon[36248]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:17.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:16 smithi145 ceph-mon[32151]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:17.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266734 got 227633266738 for osd.7 2024-03-28T18:37:17.524 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:17.986 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-03-28T18:37:17.986 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:18.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:18 smithi192 ceph-mon[36589]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:18.918 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:18 smithi145 ceph-mon[32151]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:18.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:18 smithi164 ceph-mon[36248]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:19.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 -- 172.21.15.145:0/3857482145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc010ec40 msgr2=0x7fadc0111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:19.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3857482145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc010ec40 0x7fadc0111030 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fadb800b0a0 tx=0x7fadb800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 -- 172.21.15.145:0/3857482145 shutdown_connections 2024-03-28T18:37:19.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3857482145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc010ec40 0x7fadc0111030 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3857482145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc0069e40 0x7fadc006a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3857482145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc006a760 0x7fadc010e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.397+0000 7fadc7aff700 1 -- 172.21.15.145:0/3857482145 >> 172.21.15.145:0/3857482145 conn(0x7fadc0077fc0 msgr2=0x7fadc00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- 172.21.15.145:0/3857482145 shutdown_connections 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- 172.21.15.145:0/3857482145 wait complete. 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 Processor -- start 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- start start 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 0x7fadc01a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 0x7fadc019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc01146a0 con 0x7fadc010ec40 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0114520 con 0x7fadc0069e40 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.398+0000 7fadc7aff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0114820 con 0x7fadc006a760 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc509a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 0x7fadc01a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:19.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44018/0 (socket says 172.21.15.145:44018) 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 -- 172.21.15.145:0/3222687812 learned_addr learned my addr 172.21.15.145:0/3222687812 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc609c700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 0x7fadc019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 -- 172.21.15.145:0/3222687812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 msgr2=0x7fadc01a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 0x7fadc01a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 -- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 msgr2=0x7fadc019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 0x7fadc019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 -- 172.21.15.145:0/3222687812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc019b160 con 0x7fadc0069e40 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc509a700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 0x7fadc01a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc609c700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 0x7fadc019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc589b700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00e9c0 tx=0x7fadbc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00cbd0 con 0x7fadc0069e40 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadbc00cd70 con 0x7fadc0069e40 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc010430 con 0x7fadc0069e40 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc019b450 con 0x7fadc0069e40 2024-03-28T18:37:19.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.399+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc01a7710 con 0x7fadc0069e40 2024-03-28T18:37:19.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.401+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc0068e50 con 0x7fadc0069e40 2024-03-28T18:37:19.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.402+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fadbc004750 con 0x7fadc0069e40 2024-03-28T18:37:19.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.403+0000 7fadb6ffd700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 0x7fadac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:19.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.403+0000 7fadc509a700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 0x7fadac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:19.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.403+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fadbc0cf190 con 0x7fadc0069e40 2024-03-28T18:37:19.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.404+0000 7fadc509a700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 0x7fadac0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fadc0102680 tx=0x7fadb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:19.407 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.407+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fadbc09bd70 con 0x7fadc0069e40 2024-03-28T18:37:19.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.618+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fadc019bf30 con 0x7fadc0069e40 2024-03-28T18:37:19.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.619+0000 7fadb6ffd700 1 -- 172.21.15.145:0/3222687812 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fadbc09b670 con 0x7fadc0069e40 2024-03-28T18:37:19.619 INFO:teuthology.orchestra.run.smithi145.stdout:352187318302 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 msgr2=0x7fadac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 0x7fadac0a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fadc0102680 tx=0x7fadb0009260 comp rx=0 tx=0).stop 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 msgr2=0x7fadc01a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00e9c0 tx=0x7fadbc00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 shutdown_connections 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fadac0a29a0 0x7fadac0a4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc010ec40 0x7fadc019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.621 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0069e40 0x7fadc01a0840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 --2- 172.21.15.145:0/3222687812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadc006a760 0x7fadc01a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:19.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 >> 172.21.15.145:0/3222687812 conn(0x7fadc0077fc0 msgr2=0x7fadc0110150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:19.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 shutdown_connections 2024-03-28T18:37:19.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:19.621+0000 7fadc7aff700 1 -- 172.21.15.145:0/3222687812 wait complete. 2024-03-28T18:37:19.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:19 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3222687812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:37:19.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:19 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3222687812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:37:20.022 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:19 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3222687812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:37:20.221 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 -- 172.21.15.145:0/507762757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101d30 msgr2=0x7f289c10df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.221 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/507762757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101d30 0x7f289c10df50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f289400b0d0 tx=0x7f289400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 -- 172.21.15.145:0/507762757 shutdown_connections 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/507762757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101d30 0x7f289c10df50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/507762757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c1071c0 0x7f289c1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/507762757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c101370 0x7f289c1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 -- 172.21.15.145:0/507762757 >> 172.21.15.145:0/507762757 conn(0x7f289c077f50 msgr2=0x7f289c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 -- 172.21.15.145:0/507762757 shutdown_connections 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.221+0000 7f28a2a4f700 1 -- 172.21.15.145:0/507762757 wait complete. 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 Processor -- start 2024-03-28T18:37:20.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 -- start start 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c101d30 0x7f289c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 0x7f289c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c111390 con 0x7f289c101370 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c111210 con 0x7f289c1071c0 2024-03-28T18:37:20.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a2a4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289c111510 con 0x7f289c101d30 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37408/0 (socket says 172.21.15.145:37408) 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 -- 172.21.15.145:0/2197195439 learned_addr learned my addr 172.21.15.145:0/2197195439 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289b7fe700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c101d30 0x7f289c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a0fec700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 0x7f289c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 -- 172.21.15.145:0/2197195439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c101d30 msgr2=0x7f289c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c101d30 0x7f289c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 -- 172.21.15.145:0/2197195439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 msgr2=0x7f289c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 0x7f289c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f289bfff700 1 -- 172.21.15.145:0/2197195439 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c198f40 con 0x7f289c101370 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.222+0000 7f28a0fec700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 0x7f289c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:20.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f289bfff700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f288c00e9c0 tx=0x7f288c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c00cbd0 con 0x7f289c101370 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c1991d0 con 0x7f289c101370 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288c00ee60 con 0x7f289c101370 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c005630 con 0x7f289c101370 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.223+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c102e20 con 0x7f289c101370 2024-03-28T18:37:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.224+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c199540 con 0x7f289c101370 2024-03-28T18:37:20.227 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.225+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f288c004750 con 0x7f289c101370 2024-03-28T18:37:20.227 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.226+0000 7f28997fa700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 0x7f28840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.227 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.226+0000 7f289b7fe700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 0x7f28840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.227 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.226+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f288c0cf370 con 0x7f289c101370 2024-03-28T18:37:20.227 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.226+0000 7f289b7fe700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 0x7f28840a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2890009680 tx=0x7f2890009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:20.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.233+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f288c053740 con 0x7f289c101370 2024-03-28T18:37:20.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.532+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f289c199bc0 con 0x7f289c101370 2024-03-28T18:37:20.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.532+0000 7f28997fa700 1 -- 172.21.15.145:0/2197195439 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f289c199bc0 con 0x7f289c101370 2024-03-28T18:37:20.532 INFO:teuthology.orchestra.run.smithi145.stdout:34359738454 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 msgr2=0x7f28840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 0x7f28840a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2890009680 tx=0x7f2890009260 comp rx=0 tx=0).stop 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 msgr2=0x7f289c19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f288c00e9c0 tx=0x7f288c00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 shutdown_connections 2024-03-28T18:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f28840a29a0 0x7f28840a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f289c1071c0 0x7f289c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f289c101370 0x7f289c19e590 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 --2- 172.21.15.145:0/2197195439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f289c101d30 0x7f289c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 >> 172.21.15.145:0/2197195439 conn(0x7f289c077f50 msgr2=0x7f289c06a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 shutdown_connections 2024-03-28T18:37:20.535 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.534+0000 7f28a2a4f700 1 -- 172.21.15.145:0/2197195439 wait complete. 2024-03-28T18:37:20.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/603221720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806b2e0 msgr2=0x7fcf98106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/603221720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806b2e0 0x7fcf98106dc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009990 tx=0x7fcf8c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/603221720 shutdown_connections 2024-03-28T18:37:20.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/603221720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 0x7fcf9810f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/603221720 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806a930 0x7fcf9806ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/603221720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806b2e0 0x7fcf98106dc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/603221720 >> 172.21.15.145:0/603221720 conn(0x7fcf980770e0 msgr2=0x7fcf980774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:20.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/603221720 shutdown_connections 2024-03-28T18:37:20.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.556+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/603221720 wait complete. 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 Processor -- start 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 -- start start 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 0x7fcf981016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 0x7fcf98101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98112440 con 0x7fcf9806b2e0 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981122c0 con 0x7fcf9806a930 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.557+0000 7fcf9dfc5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981125c0 con 0x7fcf981074a0 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf97fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 0x7fcf98101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44052/0 (socket says 172.21.15.145:44052) 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 -- 172.21.15.145:0/3354391332 learned_addr learned my addr 172.21.15.145:0/3354391332 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf96ffd700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 0x7fcf981016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 -- 172.21.15.145:0/3354391332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 msgr2=0x7fcf98101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 0x7fcf98101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 -- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 msgr2=0x7fcf981016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 0x7fcf981016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 -- 172.21.15.145:0/3354391332 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf981a6e40 con 0x7fcf9806a930 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf96ffd700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 0x7fcf981016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf977fe700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9a0 tx=0x7fcf8800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800cbd0 con 0x7fcf9806a930 2024-03-28T18:37:20.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8800cd70 con 0x7fcf9806a930 2024-03-28T18:37:20.561 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf981a7130 con 0x7fcf9806a930 2024-03-28T18:37:20.561 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.558+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88018750 con 0x7fcf9806a930 2024-03-28T18:37:20.561 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.559+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf981a7670 con 0x7fcf9806a930 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.559+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98068e50 con 0x7fcf9806a930 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.560+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fcf88010430 con 0x7fcf9806a930 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.561+0000 7fcf94ff9700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 0x7fcf800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.561+0000 7fcf96ffd700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 0x7fcf800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.561+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcf880cf210 con 0x7fcf9806a930 2024-03-28T18:37:20.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.562+0000 7fcf96ffd700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 0x7fcf800a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf98108870 tx=0x7fcf8c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:20.565 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.565+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf8809bdf0 con 0x7fcf9806a930 2024-03-28T18:37:20.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.777+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcf9802dbf0 con 0x7fcf9806a930 2024-03-28T18:37:20.777 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.777+0000 7fcf94ff9700 1 -- 172.21.15.145:0/3354391332 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcf8809b6f0 con 0x7fcf9806a930 2024-03-28T18:37:20.778 INFO:teuthology.orchestra.run.smithi145.stdout:141733920835 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 msgr2=0x7fcf800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 0x7fcf800a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf98108870 tx=0x7fcf8c01a040 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 msgr2=0x7fcf98101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9a0 tx=0x7fcf8800ecb0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 shutdown_connections 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf800a2920 0x7fcf800a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf981074a0 0x7fcf98101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf9806a930 0x7fcf98101190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 --2- 172.21.15.145:0/3354391332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9806b2e0 0x7fcf981016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 >> 172.21.15.145:0/3354391332 conn(0x7fcf980770e0 msgr2=0x7fcf980ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:20.780 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 shutdown_connections 2024-03-28T18:37:20.781 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.780+0000 7fcf9dfc5700 1 -- 172.21.15.145:0/3354391332 wait complete. 2024-03-28T18:37:20.878 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:20 smithi145 ceph-mon[32151]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:20.878 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:20 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2197195439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:37:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:20 smithi192 ceph-mon[36589]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:20.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:20 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2197195439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:37:20.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:20 smithi164 ceph-mon[36248]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:20.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:20 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2197195439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:37:21.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 -- 172.21.15.145:0/3782045232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 msgr2=0x7fb1141085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 --2- 172.21.15.145:0/3782045232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb1141085c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb1040099c0 tx=0x7fb104009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 -- 172.21.15.145:0/3782045232 shutdown_connections 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 --2- 172.21.15.145:0/3782045232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb11406a7e0 0x7fb11410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 --2- 172.21.15.145:0/3782045232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb1141085c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 --2- 172.21.15.145:0/3782045232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb114069e40 0x7fb11406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.985+0000 7fb11973f700 1 -- 172.21.15.145:0/3782045232 >> 172.21.15.145:0/3782045232 conn(0x7fb114077fc0 msgr2=0x7fb1140783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.986+0000 7fb11973f700 1 -- 172.21.15.145:0/3782045232 shutdown_connections 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.986+0000 7fb11973f700 1 -- 172.21.15.145:0/3782045232 wait complete. 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.986+0000 7fb11973f700 1 Processor -- start 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.986+0000 7fb11973f700 1 -- start start 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.986+0000 7fb11973f700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 0x7fb114101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb11406a7e0 0x7fb1141016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114112420 con 0x7fb1141081e0 2024-03-28T18:37:21.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141122a0 con 0x7fb114069e40 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141125a0 con 0x7fb11406a7e0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb112ffd700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 0x7fb114101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:37436/0 (socket says 172.21.15.145:37436) 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 -- 172.21.15.145:0/126976209 learned_addr learned my addr 172.21.15.145:0/126976209 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1127fc700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb11406a7e0 0x7fb1141016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 -- 172.21.15.145:0/126976209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb11406a7e0 msgr2=0x7fb1141016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb11406a7e0 0x7fb1141016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 -- 172.21.15.145:0/126976209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 msgr2=0x7fb114101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 0x7fb114101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 -- 172.21.15.145:0/126976209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1141a6de0 con 0x7fb1141081e0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb112ffd700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 0x7fb114101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb1137fe700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007ee0 tx=0x7fb10c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c004bb0 con 0x7fb1141081e0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1141a70d0 con 0x7fb1141081e0 2024-03-28T18:37:21.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb10c004d50 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c005960 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.987+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1141a75c0 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.989+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb114068e50 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.990+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb10c01a020 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.990+0000 7fb0fbfff700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 0x7fb0fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.990+0000 7fb112ffd700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 0x7fb0fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.991+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb10c0ce630 con 0x7fb1141081e0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.991+0000 7fb112ffd700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 0x7fb0fc0a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb104000c00 tx=0x7fb104005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:21.005 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:20.994+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb10c010070 con 0x7fb1141081e0 2024-03-28T18:37:21.200 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.200+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb11402da30 con 0x7fb1141081e0 2024-03-28T18:37:21.200 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.201+0000 7fb0fbfff700 1 -- 172.21.15.145:0/126976209 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb10c09b210 con 0x7fb1141081e0 2024-03-28T18:37:21.201 INFO:teuthology.orchestra.run.smithi145.stdout:107374182470 2024-03-28T18:37:21.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 msgr2=0x7fb0fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:21.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 0x7fb0fc0a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb104000c00 tx=0x7fb104005fb0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 msgr2=0x7fb114101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007ee0 tx=0x7fb10c00d3b0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 shutdown_connections 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb0fc0a2920 0x7fb0fc0a4de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb1141081e0 0x7fb114101be0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb114069e40 0x7fb114101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 --2- 172.21.15.145:0/126976209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb11406a7e0 0x7fb1141016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 >> 172.21.15.145:0/126976209 conn(0x7fb114077fc0 msgr2=0x7fb1140ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 shutdown_connections 2024-03-28T18:37:21.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:21.203+0000 7fb11973f700 1 -- 172.21.15.145:0/126976209 wait complete. 2024-03-28T18:37:21.484 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318295 got 352187318302 for osd.11 2024-03-28T18:37:21.484 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:21.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:21 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3354391332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:37:21.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:21 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/126976209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:37:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:21 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3354391332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:37:21.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:21 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/126976209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:37:21.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738454 for osd.0 2024-03-28T18:37:21.963 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:22.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:21 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3354391332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:37:22.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:21 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/126976209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:37:22.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:22 smithi164 ceph-mon[36248]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:23.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:22 smithi145 ceph-mon[32151]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:23.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920835 for osd.4 2024-03-28T18:37:23.097 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:23.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:22 smithi192 ceph-mon[36589]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 -- 172.21.15.145:0/2339397506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 msgr2=0x7f8d141027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 --2- 172.21.15.145:0/2339397506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d141027b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0099c0 tx=0x7f8d0c009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 -- 172.21.15.145:0/2339397506 shutdown_connections 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 --2- 172.21.15.145:0/2339397506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d141074b0 0x7f8d1410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 --2- 172.21.15.145:0/2339397506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d141027b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 --2- 172.21.15.145:0/2339397506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d14102d80 0x7f8d14106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 -- 172.21.15.145:0/2339397506 >> 172.21.15.145:0/2339397506 conn(0x7f8d140fdc40 msgr2=0x7f8d14100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 -- 172.21.15.145:0/2339397506 shutdown_connections 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.276+0000 7f8d191bc700 1 -- 172.21.15.145:0/2339397506 wait complete. 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 Processor -- start 2024-03-28T18:37:23.280 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 -- start start 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 0x7f8d1419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 0x7f8d14198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d141124b0 con 0x7f8d141023d0 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14112330 con 0x7f8d14102d80 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d191bc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d14112630 con 0x7f8d141074b0 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d12d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 0x7f8d1419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d12d9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:47386/0 (socket says 172.21.15.145:47386) 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d12d9d700 1 -- 172.21.15.145:0/4239454686 learned_addr learned my addr 172.21.15.145:0/4239454686 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1359e700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 0x7f8d14198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 -- 172.21.15.145:0/4239454686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 msgr2=0x7f8d14198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 0x7f8d14198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.281 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 -- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 msgr2=0x7f8d1419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1259c700 1 -- 172.21.15.145:0/4239454686 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14199030 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d12d9d700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.277+0000 7f8d1359e700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 0x7f8d14198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d1259c700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 0x7f8d1419ebc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00ef90 tx=0x7f8cfc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc0092e0 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cfc004bb0 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d141992c0 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc004280 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.278+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d141a5430 con 0x7f8d14102d80 2024-03-28T18:37:23.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.279+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d141038b0 con 0x7f8d14102d80 2024-03-28T18:37:23.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.280+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8cfc01b030 con 0x7f8d14102d80 2024-03-28T18:37:23.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.281+0000 7f8d0bfff700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 0x7f8d000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:23.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.281+0000 7f8d12d9d700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 0x7f8d000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:23.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.281+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8cfc0cf580 con 0x7f8d14102d80 2024-03-28T18:37:23.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.281+0000 7f8d12d9d700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 0x7f8d000a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c003800 tx=0x7f8d0c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:23.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.284+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8cfc022020 con 0x7f8d14102d80 2024-03-28T18:37:23.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.555+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8d1402da30 con 0x7f8d14102d80 2024-03-28T18:37:23.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.556+0000 7f8d0bfff700 1 -- 172.21.15.145:0/4239454686 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8cfc010040 con 0x7f8d14102d80 2024-03-28T18:37:23.556 INFO:teuthology.orchestra.run.smithi145.stdout:201863462969 2024-03-28T18:37:23.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 msgr2=0x7f8d000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:23.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 0x7f8d000a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c003800 tx=0x7f8d0c005960 comp rx=0 tx=0).stop 2024-03-28T18:37:23.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 msgr2=0x7f8d1419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:23.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 0x7f8d1419ebc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00ef90 tx=0x7f8cfc00c3a0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 shutdown_connections 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f8d000a2920 0x7f8d000a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d141074b0 0x7f8d14198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d141023d0 0x7f8d1419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 --2- 172.21.15.145:0/4239454686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d14102d80 0x7f8d1419ebc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 >> 172.21.15.145:0/4239454686 conn(0x7f8d140fdc40 msgr2=0x7f8d140ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 shutdown_connections 2024-03-28T18:37:23.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:23.558+0000 7f8d191bc700 1 -- 172.21.15.145:0/4239454686 wait complete. 2024-03-28T18:37:23.889 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182470 for osd.3 2024-03-28T18:37:23.890 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4239454686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:24.564 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.589 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462964 got 201863462969 for osd.6 2024-03-28T18:37:24.589 DEBUG:teuthology.parallel:result is None 2024-03-28T18:37:24.590 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T18:37:24.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4239454686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:24.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4239454686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:24.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:37:24.846 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:26.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:26 smithi192 ceph-mon[36589]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T18:37:26.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:26 smithi192 ceph-mon[36589]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:26.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:26 smithi145 ceph-mon[32151]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T18:37:26.661 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:26 smithi145 ceph-mon[32151]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/579625164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c081081e0 msgr2=0x7f0c081085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/579625164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c081081e0 0x7f0c081085c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c040099c0 tx=0x7f0c04009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/579625164 shutdown_connections 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/579625164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c0806a7e0 0x7f0c0810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/579625164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c081081e0 0x7f0c081085c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/579625164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c08069e40 0x7f0c0806a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/579625164 >> 172.21.15.145:0/579625164 conn(0x7f0c08077fc0 msgr2=0x7f0c080783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:26.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/579625164 shutdown_connections 2024-03-28T18:37:26.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.662+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/579625164 wait complete. 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 Processor -- start 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 -- start start 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c08069e40 0x7f0c0819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c081081e0 0x7f0c081986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:26.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c08112420 con 0x7f0c0806a7e0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c081122a0 con 0x7f0c081081e0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0f7d2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c081125a0 con 0x7f0c08069e40 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:47416/0 (socket says 172.21.15.145:47416) 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 -- 172.21.15.145:0/612802360 learned_addr learned my addr 172.21.15.145:0/612802360 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0d56e700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c08069e40 0x7f0c0819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0dd6f700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c081081e0 0x7f0c081986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 -- 172.21.15.145:0/612802360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c08069e40 msgr2=0x7f0c0819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c08069e40 0x7f0c0819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 -- 172.21.15.145:0/612802360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c081081e0 msgr2=0x7f0c081986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c081081e0 0x7f0c081986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.663+0000 7f0c0cd6d700 1 -- 172.21.15.145:0/612802360 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08198fa0 con 0x7f0c0806a7e0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0c0cd6d700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e9a0 tx=0x7f0bf800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf800ca10 con 0x7f0c0806a7e0 2024-03-28T18:37:26.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bf800cbb0 con 0x7f0c0806a7e0 2024-03-28T18:37:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c08199290 con 0x7f0c0806a7e0 2024-03-28T18:37:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8018680 con 0x7f0c0806a7e0 2024-03-28T18:37:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.664+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c081a5570 con 0x7f0c0806a7e0 2024-03-28T18:37:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.665+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c08103890 con 0x7f0c0806a7e0 2024-03-28T18:37:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.666+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0bf8010430 con 0x7f0c0806a7e0 2024-03-28T18:37:26.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.667+0000 7f0bfe7fc700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 0x7f0bf40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:26.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.667+0000 7f0c0d56e700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 0x7f0bf40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:26.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.667+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0bf80cf490 con 0x7f0c0806a7e0 2024-03-28T18:37:26.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.667+0000 7f0c0d56e700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 0x7f0bf40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0c04000c00 tx=0x7f0c04005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:26.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.671+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0bf809c1d0 con 0x7f0c0806a7e0 2024-03-28T18:37:26.711 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:26 smithi164 ceph-mon[36248]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T18:37:26.711 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:26 smithi164 ceph-mon[36248]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:26.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.967+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c08199bf0 con 0x7f0bf40a2890 2024-03-28T18:37:26.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.979+0000 7f0bfe7fc700 1 -- 172.21.15.145:0/612802360 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340856 (secure 0 0 0) 0x7f0c08199bf0 con 0x7f0bf40a2890 2024-03-28T18:37:26.984 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 msgr2=0x7f0bf40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 0x7f0bf40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0c04000c00 tx=0x7f0c04005fb0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 msgr2=0x7f0c0819eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e9a0 tx=0x7f0bf800ecb0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 shutdown_connections 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0bf40a2890 0x7f0bf40a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c081081e0 0x7f0c081986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c08069e40 0x7f0c0819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 --2- 172.21.15.145:0/612802360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c0806a7e0 0x7f0c0819eb30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 >> 172.21.15.145:0/612802360 conn(0x7f0c08077fc0 msgr2=0x7f0c080ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:26.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 shutdown_connections 2024-03-28T18:37:26.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:26.983+0000 7f0c0f7d2700 1 -- 172.21.15.145:0/612802360 wait complete. 2024-03-28T18:37:26.987 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-28T18:37:27.854 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-28T18:37:25.497813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658707,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":825,"num_read_kb":549,"num_write":426,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182018,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":642,"ondisk_log_size":642,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370088,"kb_used_data":12572,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571736,"statfs":{"total":1151940427776,"available":1151561457664,"internally_reserved":0,"allocated":12873728,"data_stored":8080022,"data_compressed":358785,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":55,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.523470+0000","last_change":"2024-03-28T18:35:50.713920+0000","last_active":"2024-03-28T18:37:23.523470+0000","last_peered":"2024-03-28T18:37:23.523470+0000","last_clean":"2024-03-28T18:37:23.523470+0000","last_became_active":"2024-03-28T18:35:50.712723+0000","last_became_peered":"2024-03-28T18:35:50.712723+0000","last_unstale":"2024-03-28T18:37:23.523470+0000","last_undegraded":"2024-03-28T18:37:23.523470+0000","last_fullsized":"2024-03-28T18:37:23.523470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:31.562072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594096+0000","last_change":"2024-03-28T18:35:50.714271+0000","last_active":"2024-03-28T18:36:54.594096+0000","last_peered":"2024-03-28T18:36:54.594096+0000","last_clean":"2024-03-28T18:36:54.594096+0000","last_became_active":"2024-03-28T18:35:50.713461+0000","last_became_peered":"2024-03-28T18:35:50.713461+0000","last_unstale":"2024-03-28T18:36:54.594096+0000","last_undegraded":"2024-03-28T18:36:54.594096+0000","last_fullsized":"2024-03-28T18:36:54.594096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:03:47.789033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594153+0000","last_change":"2024-03-28T18:35:50.720511+0000","last_active":"2024-03-28T18:36:54.594153+0000","last_peered":"2024-03-28T18:36:54.594153+0000","last_clean":"2024-03-28T18:36:54.594153+0000","last_became_active":"2024-03-28T18:35:50.720227+0000","last_became_peered":"2024-03-28T18:35:50.720227+0000","last_unstale":"2024-03-28T18:36:54.594153+0000","last_undegraded":"2024-03-28T18:36:54.594153+0000","last_fullsized":"2024-03-28T18:36:54.594153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:19.849187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.489081+0000","last_change":"2024-03-28T18:35:50.707852+0000","last_active":"2024-03-28T18:37:23.489081+0000","last_peered":"2024-03-28T18:37:23.489081+0000","last_clean":"2024-03-28T18:37:23.489081+0000","last_became_active":"2024-03-28T18:35:50.705465+0000","last_became_peered":"2024-03-28T18:35:50.705465+0000","last_unstale":"2024-03-28T18:37:23.489081+0000","last_undegraded":"2024-03-28T18:37:23.489081+0000","last_fullsized":"2024-03-28T18:37:23.489081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:00.251387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592800+0000","last_change":"2024-03-28T18:35:50.716142+0000","last_active":"2024-03-28T18:36:54.592800+0000","last_peered":"2024-03-28T18:36:54.592800+0000","last_clean":"2024-03-28T18:36:54.592800+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:36:54.592800+0000","last_undegraded":"2024-03-28T18:36:54.592800+0000","last_fullsized":"2024-03-28T18:36:54.592800+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:46.515184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.602406+0000","last_change":"2024-03-28T18:36:54.602406+0000","last_active":"2024-03-28T18:36:54.602406+0000","last_peered":"2024-03-28T18:36:54.602406+0000","last_clean":"2024-03-28T18:36:54.602406+0000","last_became_active":"2024-03-28T18:36:54.602016+0000","last_became_peered":"2024-03-28T18:36:54.602016+0000","last_unstale":"2024-03-28T18:36:54.602406+0000","last_undegraded":"2024-03-28T18:36:54.602406+0000","last_fullsized":"2024-03-28T18:36:54.602406+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:24:21.352607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.501388+0000","last_change":"2024-03-28T18:35:50.713781+0000","last_active":"2024-03-28T18:37:23.501388+0000","last_peered":"2024-03-28T18:37:23.501388+0000","last_clean":"2024-03-28T18:37:23.501388+0000","last_became_active":"2024-03-28T18:35:50.713400+0000","last_became_peered":"2024-03-28T18:35:50.713400+0000","last_unstale":"2024-03-28T18:37:23.501388+0000","last_undegraded":"2024-03-28T18:37:23.501388+0000","last_fullsized":"2024-03-28T18:37:23.501388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:37:21.368701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590647+0000","last_change":"2024-03-28T18:35:50.704912+0000","last_active":"2024-03-28T18:36:54.590647+0000","last_peered":"2024-03-28T18:36:54.590647+0000","last_clean":"2024-03-28T18:36:54.590647+0000","last_became_active":"2024-03-28T18:35:50.704590+0000","last_became_peered":"2024-03-28T18:35:50.704590+0000","last_unstale":"2024-03-28T18:36:54.590647+0000","last_undegraded":"2024-03-28T18:36:54.590647+0000","last_fullsized":"2024-03-28T18:36:54.590647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:17.222377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.510794+0000","last_change":"2024-03-28T18:35:50.710448+0000","last_active":"2024-03-28T18:37:23.510794+0000","last_peered":"2024-03-28T18:37:23.510794+0000","last_clean":"2024-03-28T18:37:23.510794+0000","last_became_active":"2024-03-28T18:35:50.708630+0000","last_became_peered":"2024-03-28T18:35:50.708630+0000","last_unstale":"2024-03-28T18:37:23.510794+0000","last_undegraded":"2024-03-28T18:37:23.510794+0000","last_fullsized":"2024-03-28T18:37:23.510794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:01.711677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591586+0000","last_change":"2024-03-28T18:35:50.711851+0000","last_active":"2024-03-28T18:36:54.591586+0000","last_peered":"2024-03-28T18:36:54.591586+0000","last_clean":"2024-03-28T18:36:54.591586+0000","last_became_active":"2024-03-28T18:35:50.711497+0000","last_became_peered":"2024-03-28T18:35:50.711497+0000","last_unstale":"2024-03-28T18:36:54.591586+0000","last_undegraded":"2024-03-28T18:36:54.591586+0000","last_fullsized":"2024-03-28T18:36:54.591586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:32.032471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503776+0000","last_change":"2024-03-28T18:35:50.722741+0000","last_active":"2024-03-28T18:37:23.503776+0000","last_peered":"2024-03-28T18:37:23.503776+0000","last_clean":"2024-03-28T18:37:23.503776+0000","last_became_active":"2024-03-28T18:35:50.722367+0000","last_became_peered":"2024-03-28T18:35:50.722367+0000","last_unstale":"2024-03-28T18:37:23.503776+0000","last_undegraded":"2024-03-28T18:37:23.503776+0000","last_fullsized":"2024-03-28T18:37:23.503776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:28.898030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.489866+0000","last_change":"2024-03-28T18:35:50.706167+0000","last_active":"2024-03-28T18:37:23.489866+0000","last_peered":"2024-03-28T18:37:23.489866+0000","last_clean":"2024-03-28T18:37:23.489866+0000","last_became_active":"2024-03-28T18:35:50.705485+0000","last_became_peered":"2024-03-28T18:35:50.705485+0000","last_unstale":"2024-03-28T18:37:23.489866+0000","last_undegraded":"2024-03-28T18:37:23.489866+0000","last_fullsized":"2024-03-28T18:37:23.489866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:57.492095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592161+0000","last_change":"2024-03-28T18:35:50.716920+0000","last_active":"2024-03-28T18:36:54.592161+0000","last_peered":"2024-03-28T18:36:54.592161+0000","last_clean":"2024-03-28T18:36:54.592161+0000","last_became_active":"2024-03-28T18:35:50.712942+0000","last_became_peered":"2024-03-28T18:35:50.712942+0000","last_unstale":"2024-03-28T18:36:54.592161+0000","last_undegraded":"2024-03-28T18:36:54.592161+0000","last_fullsized":"2024-03-28T18:36:54.592161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:22.550943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593896+0000","last_change":"2024-03-28T18:35:50.715257+0000","last_active":"2024-03-28T18:36:54.593896+0000","last_peered":"2024-03-28T18:36:54.593896+0000","last_clean":"2024-03-28T18:36:54.593896+0000","last_became_active":"2024-03-28T18:35:50.714943+0000","last_became_peered":"2024-03-28T18:35:50.714943+0000","last_unstale":"2024-03-28T18:36:54.593896+0000","last_undegraded":"2024-03-28T18:36:54.593896+0000","last_fullsized":"2024-03-28T18:36:54.593896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:39:05.683273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593612+0000","last_change":"2024-03-28T18:35:50.718042+0000","last_active":"2024-03-28T18:36:54.593612+0000","last_peered":"2024-03-28T18:36:54.593612+0000","last_clean":"2024-03-28T18:36:54.593612+0000","last_became_active":"2024-03-28T18:35:50.717713+0000","last_became_peered":"2024-03-28T18:35:50.717713+0000","last_unstale":"2024-03-28T18:36:54.593612+0000","last_undegraded":"2024-03-28T18:36:54.593612+0000","last_fullsized":"2024-03-28T18:36:54.593612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:51.764882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.531651+0000","last_change":"2024-03-28T18:35:50.720741+0000","last_active":"2024-03-28T18:37:23.531651+0000","last_peered":"2024-03-28T18:37:23.531651+0000","last_clean":"2024-03-28T18:37:23.531651+0000","last_became_active":"2024-03-28T18:35:50.720038+0000","last_became_peered":"2024-03-28T18:35:50.720038+0000","last_unstale":"2024-03-28T18:37:23.531651+0000","last_undegraded":"2024-03-28T18:37:23.531651+0000","last_fullsized":"2024-03-28T18:37:23.531651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:48.001768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.523700+0000","last_change":"2024-03-28T18:35:50.714152+0000","last_active":"2024-03-28T18:37:23.523700+0000","last_peered":"2024-03-28T18:37:23.523700+0000","last_clean":"2024-03-28T18:37:23.523700+0000","last_became_active":"2024-03-28T18:35:50.713813+0000","last_became_peered":"2024-03-28T18:35:50.713813+0000","last_unstale":"2024-03-28T18:37:23.523700+0000","last_undegraded":"2024-03-28T18:37:23.523700+0000","last_fullsized":"2024-03-28T18:37:23.523700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:48.698925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.512794+0000","last_change":"2024-03-28T18:35:50.708046+0000","last_active":"2024-03-28T18:37:23.512794+0000","last_peered":"2024-03-28T18:37:23.512794+0000","last_clean":"2024-03-28T18:37:23.512794+0000","last_became_active":"2024-03-28T18:35:50.707694+0000","last_became_peered":"2024-03-28T18:35:50.707694+0000","last_unstale":"2024-03-28T18:37:23.512794+0000","last_undegraded":"2024-03-28T18:37:23.512794+0000","last_fullsized":"2024-03-28T18:37:23.512794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:37:46.494020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514980+0000","last_change":"2024-03-28T18:35:50.714366+0000","last_active":"2024-03-28T18:37:23.514980+0000","last_peered":"2024-03-28T18:37:23.514980+0000","last_clean":"2024-03-28T18:37:23.514980+0000","last_became_active":"2024-03-28T18:35:50.714046+0000","last_became_peered":"2024-03-28T18:35:50.714046+0000","last_unstale":"2024-03-28T18:37:23.514980+0000","last_undegraded":"2024-03-28T18:37:23.514980+0000","last_fullsized":"2024-03-28T18:37:23.514980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:24.757304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595794+0000","last_change":"2024-03-28T18:35:55.032307+0000","last_active":"2024-03-28T18:36:54.595794+0000","last_peered":"2024-03-28T18:36:54.595794+0000","last_clean":"2024-03-28T18:36:54.595794+0000","last_became_active":"2024-03-28T18:35:55.031787+0000","last_became_peered":"2024-03-28T18:35:55.031787+0000","last_unstale":"2024-03-28T18:36:54.595794+0000","last_undegraded":"2024-03-28T18:36:54.595794+0000","last_fullsized":"2024-03-28T18:36:54.595794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:15.649053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":113,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.113327+0000","last_change":"2024-03-28T18:35:49.933942+0000","last_active":"2024-03-28T18:37:23.113327+0000","last_peered":"2024-03-28T18:37:23.113327+0000","last_clean":"2024-03-28T18:37:23.113327+0000","last_became_active":"2024-03-28T18:35:49.933608+0000","last_became_peered":"2024-03-28T18:35:49.933608+0000","last_unstale":"2024-03-28T18:37:23.113327+0000","last_undegraded":"2024-03-28T18:37:23.113327+0000","last_fullsized":"2024-03-28T18:37:23.113327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:24.327806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591592+0000","last_change":"2024-03-28T18:35:50.704543+0000","last_active":"2024-03-28T18:36:54.591592+0000","last_peered":"2024-03-28T18:36:54.591592+0000","last_clean":"2024-03-28T18:36:54.591592+0000","last_became_active":"2024-03-28T18:35:50.704232+0000","last_became_peered":"2024-03-28T18:35:50.704232+0000","last_unstale":"2024-03-28T18:36:54.591592+0000","last_undegraded":"2024-03-28T18:36:54.591592+0000","last_fullsized":"2024-03-28T18:36:54.591592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:30.934620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589094+0000","last_change":"2024-03-28T18:35:55.032545+0000","last_active":"2024-03-28T18:36:54.589094+0000","last_peered":"2024-03-28T18:36:54.589094+0000","last_clean":"2024-03-28T18:36:54.589094+0000","last_became_active":"2024-03-28T18:35:55.032011+0000","last_became_peered":"2024-03-28T18:35:55.032011+0000","last_unstale":"2024-03-28T18:36:54.589094+0000","last_undegraded":"2024-03-28T18:36:54.589094+0000","last_fullsized":"2024-03-28T18:36:54.589094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:46.960816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591842+0000","last_change":"2024-03-28T18:35:49.940627+0000","last_active":"2024-03-28T18:36:54.591842+0000","last_peered":"2024-03-28T18:36:54.591842+0000","last_clean":"2024-03-28T18:36:54.591842+0000","last_became_active":"2024-03-28T18:35:49.940313+0000","last_became_peered":"2024-03-28T18:35:49.940313+0000","last_unstale":"2024-03-28T18:36:54.591842+0000","last_undegraded":"2024-03-28T18:36:54.591842+0000","last_fullsized":"2024-03-28T18:36:54.591842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:24.215713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594371+0000","last_change":"2024-03-28T18:35:50.720053+0000","last_active":"2024-03-28T18:36:54.594371+0000","last_peered":"2024-03-28T18:36:54.594371+0000","last_clean":"2024-03-28T18:36:54.594371+0000","last_became_active":"2024-03-28T18:35:50.719698+0000","last_became_peered":"2024-03-28T18:35:50.719698+0000","last_unstale":"2024-03-28T18:36:54.594371+0000","last_undegraded":"2024-03-28T18:36:54.594371+0000","last_fullsized":"2024-03-28T18:36:54.594371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:30.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.598163+0000","last_change":"2024-03-28T18:36:54.598163+0000","last_active":"2024-03-28T18:36:54.598163+0000","last_peered":"2024-03-28T18:36:54.598163+0000","last_clean":"2024-03-28T18:36:54.598163+0000","last_became_active":"2024-03-28T18:36:54.597729+0000","last_became_peered":"2024-03-28T18:36:54.597729+0000","last_unstale":"2024-03-28T18:36:54.598163+0000","last_undegraded":"2024-03-28T18:36:54.598163+0000","last_fullsized":"2024-03-28T18:36:54.598163+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:04.471576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592704+0000","last_change":"2024-03-28T18:35:49.938447+0000","last_active":"2024-03-28T18:36:54.592704+0000","last_peered":"2024-03-28T18:36:54.592704+0000","last_clean":"2024-03-28T18:36:54.592704+0000","last_became_active":"2024-03-28T18:35:49.938125+0000","last_became_peered":"2024-03-28T18:35:49.938125+0000","last_unstale":"2024-03-28T18:36:54.592704+0000","last_undegraded":"2024-03-28T18:36:54.592704+0000","last_fullsized":"2024-03-28T18:36:54.592704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:31.064360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515806+0000","last_change":"2024-03-28T18:35:50.711208+0000","last_active":"2024-03-28T18:37:23.515806+0000","last_peered":"2024-03-28T18:37:23.515806+0000","last_clean":"2024-03-28T18:37:23.515806+0000","last_became_active":"2024-03-28T18:35:50.710306+0000","last_became_peered":"2024-03-28T18:35:50.710306+0000","last_unstale":"2024-03-28T18:37:23.515806+0000","last_undegraded":"2024-03-28T18:37:23.515806+0000","last_fullsized":"2024-03-28T18:37:23.515806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.449619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589142+0000","last_change":"2024-03-28T18:35:55.027968+0000","last_active":"2024-03-28T18:36:54.589142+0000","last_peered":"2024-03-28T18:36:54.589142+0000","last_clean":"2024-03-28T18:36:54.589142+0000","last_became_active":"2024-03-28T18:35:55.027486+0000","last_became_peered":"2024-03-28T18:35:55.027486+0000","last_unstale":"2024-03-28T18:36:54.589142+0000","last_undegraded":"2024-03-28T18:36:54.589142+0000","last_fullsized":"2024-03-28T18:36:54.589142+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:30.088482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592370+0000","last_change":"2024-03-28T18:35:49.936488+0000","last_active":"2024-03-28T18:36:54.592370+0000","last_peered":"2024-03-28T18:36:54.592370+0000","last_clean":"2024-03-28T18:36:54.592370+0000","last_became_active":"2024-03-28T18:35:49.936122+0000","last_became_peered":"2024-03-28T18:35:49.936122+0000","last_unstale":"2024-03-28T18:36:54.592370+0000","last_undegraded":"2024-03-28T18:36:54.592370+0000","last_fullsized":"2024-03-28T18:36:54.592370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:51.096079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.511466+0000","last_change":"2024-03-28T18:35:50.704075+0000","last_active":"2024-03-28T18:37:23.511466+0000","last_peered":"2024-03-28T18:37:23.511466+0000","last_clean":"2024-03-28T18:37:23.511466+0000","last_became_active":"2024-03-28T18:35:50.703726+0000","last_became_peered":"2024-03-28T18:35:50.703726+0000","last_unstale":"2024-03-28T18:37:23.511466+0000","last_undegraded":"2024-03-28T18:37:23.511466+0000","last_fullsized":"2024-03-28T18:37:23.511466+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:08.522098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596528+0000","last_change":"2024-03-28T18:35:55.023382+0000","last_active":"2024-03-28T18:36:54.596528+0000","last_peered":"2024-03-28T18:36:54.596528+0000","last_clean":"2024-03-28T18:36:54.596528+0000","last_became_active":"2024-03-28T18:35:55.022992+0000","last_became_peered":"2024-03-28T18:35:55.022992+0000","last_unstale":"2024-03-28T18:36:54.596528+0000","last_undegraded":"2024-03-28T18:36:54.596528+0000","last_fullsized":"2024-03-28T18:36:54.596528+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:09.260173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:22.086765+0000","last_change":"2024-03-28T18:35:49.939304+0000","last_active":"2024-03-28T18:37:22.086765+0000","last_peered":"2024-03-28T18:37:22.086765+0000","last_clean":"2024-03-28T18:37:22.086765+0000","last_became_active":"2024-03-28T18:35:49.938976+0000","last_became_peered":"2024-03-28T18:35:49.938976+0000","last_unstale":"2024-03-28T18:37:22.086765+0000","last_undegraded":"2024-03-28T18:37:22.086765+0000","last_fullsized":"2024-03-28T18:37:22.086765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:37.368257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592507+0000","last_change":"2024-03-28T18:35:50.715841+0000","last_active":"2024-03-28T18:36:54.592507+0000","last_peered":"2024-03-28T18:36:54.592507+0000","last_clean":"2024-03-28T18:36:54.592507+0000","last_became_active":"2024-03-28T18:35:50.715365+0000","last_became_peered":"2024-03-28T18:35:50.715365+0000","last_unstale":"2024-03-28T18:36:54.592507+0000","last_undegraded":"2024-03-28T18:36:54.592507+0000","last_fullsized":"2024-03-28T18:36:54.592507+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:33.009378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596592+0000","last_change":"2024-03-28T18:35:55.033073+0000","last_active":"2024-03-28T18:36:54.596592+0000","last_peered":"2024-03-28T18:36:54.596592+0000","last_clean":"2024-03-28T18:36:54.596592+0000","last_became_active":"2024-03-28T18:35:55.032649+0000","last_became_peered":"2024-03-28T18:35:55.032649+0000","last_unstale":"2024-03-28T18:36:54.596592+0000","last_undegraded":"2024-03-28T18:36:54.596592+0000","last_fullsized":"2024-03-28T18:36:54.596592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:14.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591110+0000","last_change":"2024-03-28T18:35:49.932672+0000","last_active":"2024-03-28T18:36:54.591110+0000","last_peered":"2024-03-28T18:36:54.591110+0000","last_clean":"2024-03-28T18:36:54.591110+0000","last_became_active":"2024-03-28T18:35:49.932207+0000","last_became_peered":"2024-03-28T18:35:49.932207+0000","last_unstale":"2024-03-28T18:36:54.591110+0000","last_undegraded":"2024-03-28T18:36:54.591110+0000","last_fullsized":"2024-03-28T18:36:54.591110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:50.606075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.522581+0000","last_change":"2024-03-28T18:35:50.696535+0000","last_active":"2024-03-28T18:37:23.522581+0000","last_peered":"2024-03-28T18:37:23.522581+0000","last_clean":"2024-03-28T18:37:23.522581+0000","last_became_active":"2024-03-28T18:35:50.696142+0000","last_became_peered":"2024-03-28T18:35:50.696142+0000","last_unstale":"2024-03-28T18:37:23.522581+0000","last_undegraded":"2024-03-28T18:37:23.522581+0000","last_fullsized":"2024-03-28T18:37:23.522581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:24:59.749302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596689+0000","last_change":"2024-03-28T18:36:54.596689+0000","last_active":"2024-03-28T18:36:54.596689+0000","last_peered":"2024-03-28T18:36:54.596689+0000","last_clean":"2024-03-28T18:36:54.596689+0000","last_became_active":"2024-03-28T18:36:54.595693+0000","last_became_peered":"2024-03-28T18:36:54.595693+0000","last_unstale":"2024-03-28T18:36:54.596689+0000","last_undegraded":"2024-03-28T18:36:54.596689+0000","last_fullsized":"2024-03-28T18:36:54.596689+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:20.698835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590946+0000","last_change":"2024-03-28T18:35:49.929798+0000","last_active":"2024-03-28T18:36:54.590946+0000","last_peered":"2024-03-28T18:36:54.590946+0000","last_clean":"2024-03-28T18:36:54.590946+0000","last_became_active":"2024-03-28T18:35:49.929278+0000","last_became_peered":"2024-03-28T18:35:49.929278+0000","last_unstale":"2024-03-28T18:36:54.590946+0000","last_undegraded":"2024-03-28T18:36:54.590946+0000","last_fullsized":"2024-03-28T18:36:54.590946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:33:13.078225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590601+0000","last_change":"2024-03-28T18:35:50.702935+0000","last_active":"2024-03-28T18:36:54.590601+0000","last_peered":"2024-03-28T18:36:54.590601+0000","last_clean":"2024-03-28T18:36:54.590601+0000","last_became_active":"2024-03-28T18:35:50.702586+0000","last_became_peered":"2024-03-28T18:35:50.702586+0000","last_unstale":"2024-03-28T18:36:54.590601+0000","last_undegraded":"2024-03-28T18:36:54.590601+0000","last_fullsized":"2024-03-28T18:36:54.590601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:50.335605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594102+0000","last_change":"2024-03-28T18:35:55.028541+0000","last_active":"2024-03-28T18:36:54.594102+0000","last_peered":"2024-03-28T18:36:54.594102+0000","last_clean":"2024-03-28T18:36:54.594102+0000","last_became_active":"2024-03-28T18:35:55.028142+0000","last_became_peered":"2024-03-28T18:35:55.028142+0000","last_unstale":"2024-03-28T18:36:54.594102+0000","last_undegraded":"2024-03-28T18:36:54.594102+0000","last_fullsized":"2024-03-28T18:36:54.594102+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:20.738071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588676+0000","last_change":"2024-03-28T18:35:49.938255+0000","last_active":"2024-03-28T18:36:54.588676+0000","last_peered":"2024-03-28T18:36:54.588676+0000","last_clean":"2024-03-28T18:36:54.588676+0000","last_became_active":"2024-03-28T18:35:49.937930+0000","last_became_peered":"2024-03-28T18:35:49.937930+0000","last_unstale":"2024-03-28T18:36:54.588676+0000","last_undegraded":"2024-03-28T18:36:54.588676+0000","last_fullsized":"2024-03-28T18:36:54.588676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:41.621941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.504921+0000","last_change":"2024-03-28T18:35:50.706437+0000","last_active":"2024-03-28T18:37:23.504921+0000","last_peered":"2024-03-28T18:37:23.504921+0000","last_clean":"2024-03-28T18:37:23.504921+0000","last_became_active":"2024-03-28T18:35:50.706139+0000","last_became_peered":"2024-03-28T18:35:50.706139+0000","last_unstale":"2024-03-28T18:37:23.504921+0000","last_undegraded":"2024-03-28T18:37:23.504921+0000","last_fullsized":"2024-03-28T18:37:23.504921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:52.750896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.598151+0000","last_change":"2024-03-28T18:36:54.598151+0000","last_active":"2024-03-28T18:36:54.598151+0000","last_peered":"2024-03-28T18:36:54.598151+0000","last_clean":"2024-03-28T18:36:54.598151+0000","last_became_active":"2024-03-28T18:36:54.597664+0000","last_became_peered":"2024-03-28T18:36:54.597664+0000","last_unstale":"2024-03-28T18:36:54.598151+0000","last_undegraded":"2024-03-28T18:36:54.598151+0000","last_fullsized":"2024-03-28T18:36:54.598151+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:23:34.772747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,8],"acting":[3,7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:20.084231+0000","last_change":"2024-03-28T18:35:49.940518+0000","last_active":"2024-03-28T18:37:20.084231+0000","last_peered":"2024-03-28T18:37:20.084231+0000","last_clean":"2024-03-28T18:37:20.084231+0000","last_became_active":"2024-03-28T18:35:49.940208+0000","last_became_peered":"2024-03-28T18:35:49.940208+0000","last_unstale":"2024-03-28T18:37:20.084231+0000","last_undegraded":"2024-03-28T18:37:20.084231+0000","last_fullsized":"2024-03-28T18:37:20.084231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:55.557840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515829+0000","last_change":"2024-03-28T18:35:50.710781+0000","last_active":"2024-03-28T18:37:23.515829+0000","last_peered":"2024-03-28T18:37:23.515829+0000","last_clean":"2024-03-28T18:37:23.515829+0000","last_became_active":"2024-03-28T18:35:50.710458+0000","last_became_peered":"2024-03-28T18:35:50.710458+0000","last_unstale":"2024-03-28T18:37:23.515829+0000","last_undegraded":"2024-03-28T18:37:23.515829+0000","last_fullsized":"2024-03-28T18:37:23.515829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:49.020331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601148+0000","last_change":"2024-03-28T18:35:55.034023+0000","last_active":"2024-03-28T18:36:54.601148+0000","last_peered":"2024-03-28T18:36:54.601148+0000","last_clean":"2024-03-28T18:36:54.601148+0000","last_became_active":"2024-03-28T18:35:55.033634+0000","last_became_peered":"2024-03-28T18:35:55.033634+0000","last_unstale":"2024-03-28T18:36:54.601148+0000","last_undegraded":"2024-03-28T18:36:54.601148+0000","last_fullsized":"2024-03-28T18:36:54.601148+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:41.221061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.603273+0000","last_change":"2024-03-28T18:36:54.603273+0000","last_active":"2024-03-28T18:36:54.603273+0000","last_peered":"2024-03-28T18:36:54.603273+0000","last_clean":"2024-03-28T18:36:54.603273+0000","last_became_active":"2024-03-28T18:36:54.596666+0000","last_became_peered":"2024-03-28T18:36:54.596666+0000","last_unstale":"2024-03-28T18:36:54.603273+0000","last_undegraded":"2024-03-28T18:36:54.603273+0000","last_fullsized":"2024-03-28T18:36:54.603273+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:17.480746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594372+0000","last_change":"2024-03-28T18:35:50.718480+0000","last_active":"2024-03-28T18:36:54.594372+0000","last_peered":"2024-03-28T18:36:54.594372+0000","last_clean":"2024-03-28T18:36:54.594372+0000","last_became_active":"2024-03-28T18:35:50.717253+0000","last_became_peered":"2024-03-28T18:35:50.717253+0000","last_unstale":"2024-03-28T18:36:54.594372+0000","last_undegraded":"2024-03-28T18:36:54.594372+0000","last_fullsized":"2024-03-28T18:36:54.594372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:59:30.071806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589384+0000","last_change":"2024-03-28T18:35:55.032461+0000","last_active":"2024-03-28T18:36:54.589384+0000","last_peered":"2024-03-28T18:36:54.589384+0000","last_clean":"2024-03-28T18:36:54.589384+0000","last_became_active":"2024-03-28T18:35:55.032029+0000","last_became_peered":"2024-03-28T18:35:55.032029+0000","last_unstale":"2024-03-28T18:36:54.589384+0000","last_undegraded":"2024-03-28T18:36:54.589384+0000","last_fullsized":"2024-03-28T18:36:54.589384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:27:11.216371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589446+0000","last_change":"2024-03-28T18:35:49.934357+0000","last_active":"2024-03-28T18:36:54.589446+0000","last_peered":"2024-03-28T18:36:54.589446+0000","last_clean":"2024-03-28T18:36:54.589446+0000","last_became_active":"2024-03-28T18:35:49.933994+0000","last_became_peered":"2024-03-28T18:35:49.933994+0000","last_unstale":"2024-03-28T18:36:54.589446+0000","last_undegraded":"2024-03-28T18:36:54.589446+0000","last_fullsized":"2024-03-28T18:36:54.589446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:21.408585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514262+0000","last_change":"2024-03-28T18:35:50.716771+0000","last_active":"2024-03-28T18:37:23.514262+0000","last_peered":"2024-03-28T18:37:23.514262+0000","last_clean":"2024-03-28T18:37:23.514262+0000","last_became_active":"2024-03-28T18:35:50.716393+0000","last_became_peered":"2024-03-28T18:35:50.716393+0000","last_unstale":"2024-03-28T18:37:23.514262+0000","last_undegraded":"2024-03-28T18:37:23.514262+0000","last_fullsized":"2024-03-28T18:37:23.514262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:33:10.881475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594854+0000","last_change":"2024-03-28T18:36:54.594281+0000","last_active":"2024-03-28T18:36:54.594854+0000","last_peered":"2024-03-28T18:36:54.594854+0000","last_clean":"2024-03-28T18:36:54.594854+0000","last_became_active":"2024-03-28T18:36:54.593846+0000","last_became_peered":"2024-03-28T18:36:54.593846+0000","last_unstale":"2024-03-28T18:36:54.594854+0000","last_undegraded":"2024-03-28T18:36:54.594854+0000","last_fullsized":"2024-03-28T18:36:54.594854+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:51.911167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'32","reported_seq":204,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:20.086678+0000","last_change":"2024-03-28T18:35:49.934499+0000","last_active":"2024-03-28T18:37:20.086678+0000","last_peered":"2024-03-28T18:37:20.086678+0000","last_clean":"2024-03-28T18:37:20.086678+0000","last_became_active":"2024-03-28T18:35:49.934150+0000","last_became_peered":"2024-03-28T18:35:49.934150+0000","last_unstale":"2024-03-28T18:37:20.086678+0000","last_undegraded":"2024-03-28T18:37:20.086678+0000","last_fullsized":"2024-03-28T18:37:20.086678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:53.661593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61151,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":132,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592503+0000","last_change":"2024-03-28T18:35:50.714162+0000","last_active":"2024-03-28T18:36:54.592503+0000","last_peered":"2024-03-28T18:36:54.592503+0000","last_clean":"2024-03-28T18:36:54.592503+0000","last_became_active":"2024-03-28T18:35:50.713819+0000","last_became_peered":"2024-03-28T18:35:50.713819+0000","last_unstale":"2024-03-28T18:36:54.592503+0000","last_undegraded":"2024-03-28T18:36:54.592503+0000","last_fullsized":"2024-03-28T18:36:54.592503+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:49:19.365559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589676+0000","last_change":"2024-03-28T18:35:55.028460+0000","last_active":"2024-03-28T18:36:54.589676+0000","last_peered":"2024-03-28T18:36:54.589676+0000","last_clean":"2024-03-28T18:36:54.589676+0000","last_became_active":"2024-03-28T18:35:55.028015+0000","last_became_peered":"2024-03-28T18:35:55.028015+0000","last_unstale":"2024-03-28T18:36:54.589676+0000","last_undegraded":"2024-03-28T18:36:54.589676+0000","last_fullsized":"2024-03-28T18:36:54.589676+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:00.864611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596300+0000","last_change":"2024-03-28T18:35:49.938090+0000","last_active":"2024-03-28T18:36:54.596300+0000","last_peered":"2024-03-28T18:36:54.596300+0000","last_clean":"2024-03-28T18:36:54.596300+0000","last_became_active":"2024-03-28T18:35:49.937572+0000","last_became_peered":"2024-03-28T18:35:49.937572+0000","last_unstale":"2024-03-28T18:36:54.596300+0000","last_undegraded":"2024-03-28T18:36:54.596300+0000","last_fullsized":"2024-03-28T18:36:54.596300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:41.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591114+0000","last_change":"2024-03-28T18:35:50.717286+0000","last_active":"2024-03-28T18:36:54.591114+0000","last_peered":"2024-03-28T18:36:54.591114+0000","last_clean":"2024-03-28T18:36:54.591114+0000","last_became_active":"2024-03-28T18:35:50.712917+0000","last_became_peered":"2024-03-28T18:35:50.712917+0000","last_unstale":"2024-03-28T18:36:54.591114+0000","last_undegraded":"2024-03-28T18:36:54.591114+0000","last_fullsized":"2024-03-28T18:36:54.591114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:10:42.306028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590424+0000","last_change":"2024-03-28T18:35:55.023384+0000","last_active":"2024-03-28T18:36:54.590424+0000","last_peered":"2024-03-28T18:36:54.590424+0000","last_clean":"2024-03-28T18:36:54.590424+0000","last_became_active":"2024-03-28T18:35:55.023013+0000","last_became_peered":"2024-03-28T18:35:55.023013+0000","last_unstale":"2024-03-28T18:36:54.590424+0000","last_undegraded":"2024-03-28T18:36:54.590424+0000","last_fullsized":"2024-03-28T18:36:54.590424+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:01.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589045+0000","last_change":"2024-03-28T18:35:49.939073+0000","last_active":"2024-03-28T18:36:54.589045+0000","last_peered":"2024-03-28T18:36:54.589045+0000","last_clean":"2024-03-28T18:36:54.589045+0000","last_became_active":"2024-03-28T18:35:49.938764+0000","last_became_peered":"2024-03-28T18:35:49.938764+0000","last_unstale":"2024-03-28T18:36:54.589045+0000","last_undegraded":"2024-03-28T18:36:54.589045+0000","last_fullsized":"2024-03-28T18:36:54.589045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:52.089059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.516846+0000","last_change":"2024-03-28T18:35:50.716132+0000","last_active":"2024-03-28T18:37:23.516846+0000","last_peered":"2024-03-28T18:37:23.516846+0000","last_clean":"2024-03-28T18:37:23.516846+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:37:23.516846+0000","last_undegraded":"2024-03-28T18:37:23.516846+0000","last_fullsized":"2024-03-28T18:37:23.516846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:57.930739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601576+0000","last_change":"2024-03-28T18:35:55.035553+0000","last_active":"2024-03-28T18:36:54.601576+0000","last_peered":"2024-03-28T18:36:54.601576+0000","last_clean":"2024-03-28T18:36:54.601576+0000","last_became_active":"2024-03-28T18:35:55.035134+0000","last_became_peered":"2024-03-28T18:35:55.035134+0000","last_unstale":"2024-03-28T18:36:54.601576+0000","last_undegraded":"2024-03-28T18:36:54.601576+0000","last_fullsized":"2024-03-28T18:36:54.601576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:03:52.976361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.097488+0000","last_change":"2024-03-28T18:35:49.932696+0000","last_active":"2024-03-28T18:37:23.097488+0000","last_peered":"2024-03-28T18:37:23.097488+0000","last_clean":"2024-03-28T18:37:23.097488+0000","last_became_active":"2024-03-28T18:35:49.932350+0000","last_became_peered":"2024-03-28T18:35:49.932350+0000","last_unstale":"2024-03-28T18:37:23.097488+0000","last_undegraded":"2024-03-28T18:37:23.097488+0000","last_fullsized":"2024-03-28T18:37:23.097488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:26.183549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":3,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.500849+0000","last_change":"2024-03-28T18:35:50.712734+0000","last_active":"2024-03-28T18:37:23.500849+0000","last_peered":"2024-03-28T18:37:23.500849+0000","last_clean":"2024-03-28T18:37:23.500849+0000","last_became_active":"2024-03-28T18:35:50.712409+0000","last_became_peered":"2024-03-28T18:35:50.712409+0000","last_unstale":"2024-03-28T18:37:23.500849+0000","last_undegraded":"2024-03-28T18:37:23.500849+0000","last_fullsized":"2024-03-28T18:37:23.500849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:36.344182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591837+0000","last_change":"2024-03-28T18:35:55.025047+0000","last_active":"2024-03-28T18:36:54.591837+0000","last_peered":"2024-03-28T18:36:54.591837+0000","last_clean":"2024-03-28T18:36:54.591837+0000","last_became_active":"2024-03-28T18:35:55.024652+0000","last_became_peered":"2024-03-28T18:35:55.024652+0000","last_unstale":"2024-03-28T18:36:54.591837+0000","last_undegraded":"2024-03-28T18:36:54.591837+0000","last_fullsized":"2024-03-28T18:36:54.591837+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:17.750951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.614368+0000","last_change":"2024-03-28T18:36:54.613995+0000","last_active":"2024-03-28T18:36:54.614368+0000","last_peered":"2024-03-28T18:36:54.614368+0000","last_clean":"2024-03-28T18:36:54.614368+0000","last_became_active":"2024-03-28T18:36:54.599838+0000","last_became_peered":"2024-03-28T18:36:54.599838+0000","last_unstale":"2024-03-28T18:36:54.614368+0000","last_undegraded":"2024-03-28T18:36:54.614368+0000","last_fullsized":"2024-03-28T18:36:54.614368+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:02.855413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.508762+0000","last_change":"2024-03-28T18:35:50.705023+0000","last_active":"2024-03-28T18:37:23.508762+0000","last_peered":"2024-03-28T18:37:23.508762+0000","last_clean":"2024-03-28T18:37:23.508762+0000","last_became_active":"2024-03-28T18:35:50.704694+0000","last_became_peered":"2024-03-28T18:35:50.704694+0000","last_unstale":"2024-03-28T18:37:23.508762+0000","last_undegraded":"2024-03-28T18:37:23.508762+0000","last_fullsized":"2024-03-28T18:37:23.508762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:46.677189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588854+0000","last_change":"2024-03-28T18:35:55.028311+0000","last_active":"2024-03-28T18:36:54.588854+0000","last_peered":"2024-03-28T18:36:54.588854+0000","last_clean":"2024-03-28T18:36:54.588854+0000","last_became_active":"2024-03-28T18:35:55.027914+0000","last_became_peered":"2024-03-28T18:35:55.027914+0000","last_unstale":"2024-03-28T18:36:54.588854+0000","last_undegraded":"2024-03-28T18:36:54.588854+0000","last_fullsized":"2024-03-28T18:36:54.588854+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:59.684230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593503+0000","last_change":"2024-03-28T18:35:49.938004+0000","last_active":"2024-03-28T18:36:54.593503+0000","last_peered":"2024-03-28T18:36:54.593503+0000","last_clean":"2024-03-28T18:36:54.593503+0000","last_became_active":"2024-03-28T18:35:49.937615+0000","last_became_peered":"2024-03-28T18:35:49.937615+0000","last_unstale":"2024-03-28T18:36:54.593503+0000","last_undegraded":"2024-03-28T18:36:54.593503+0000","last_fullsized":"2024-03-28T18:36:54.593503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:21.619857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503409+0000","last_change":"2024-03-28T18:35:50.719276+0000","last_active":"2024-03-28T18:37:23.503409+0000","last_peered":"2024-03-28T18:37:23.503409+0000","last_clean":"2024-03-28T18:37:23.503409+0000","last_became_active":"2024-03-28T18:35:50.718896+0000","last_became_peered":"2024-03-28T18:35:50.718896+0000","last_unstale":"2024-03-28T18:37:23.503409+0000","last_undegraded":"2024-03-28T18:37:23.503409+0000","last_fullsized":"2024-03-28T18:37:23.503409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:24.494713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592412+0000","last_change":"2024-03-28T18:35:55.099193+0000","last_active":"2024-03-28T18:36:54.592412+0000","last_peered":"2024-03-28T18:36:54.592412+0000","last_clean":"2024-03-28T18:36:54.592412+0000","last_became_active":"2024-03-28T18:35:55.098795+0000","last_became_peered":"2024-03-28T18:35:55.098795+0000","last_unstale":"2024-03-28T18:36:54.592412+0000","last_undegraded":"2024-03-28T18:36:54.592412+0000","last_fullsized":"2024-03-28T18:36:54.592412+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:07.872554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601410+0000","last_change":"2024-03-28T18:35:49.936885+0000","last_active":"2024-03-28T18:36:54.601410+0000","last_peered":"2024-03-28T18:36:54.601410+0000","last_clean":"2024-03-28T18:36:54.601410+0000","last_became_active":"2024-03-28T18:35:49.936563+0000","last_became_peered":"2024-03-28T18:35:49.936563+0000","last_unstale":"2024-03-28T18:36:54.601410+0000","last_undegraded":"2024-03-28T18:36:54.601410+0000","last_fullsized":"2024-03-28T18:36:54.601410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:33.631160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591288+0000","last_change":"2024-03-28T18:35:50.713750+0000","last_active":"2024-03-28T18:36:54.591288+0000","last_peered":"2024-03-28T18:36:54.591288+0000","last_clean":"2024-03-28T18:36:54.591288+0000","last_became_active":"2024-03-28T18:35:50.713378+0000","last_became_peered":"2024-03-28T18:35:50.713378+0000","last_unstale":"2024-03-28T18:36:54.591288+0000","last_undegraded":"2024-03-28T18:36:54.591288+0000","last_fullsized":"2024-03-28T18:36:54.591288+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:36.072651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591101+0000","last_change":"2024-03-28T18:35:55.030827+0000","last_active":"2024-03-28T18:36:54.591101+0000","last_peered":"2024-03-28T18:36:54.591101+0000","last_clean":"2024-03-28T18:36:54.591101+0000","last_became_active":"2024-03-28T18:35:55.030382+0000","last_became_peered":"2024-03-28T18:35:55.030382+0000","last_unstale":"2024-03-28T18:36:54.591101+0000","last_undegraded":"2024-03-28T18:36:54.591101+0000","last_fullsized":"2024-03-28T18:36:54.591101+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:41:58.371568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":104,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:22.112826+0000","last_change":"2024-03-28T18:35:49.932290+0000","last_active":"2024-03-28T18:37:22.112826+0000","last_peered":"2024-03-28T18:37:22.112826+0000","last_clean":"2024-03-28T18:37:22.112826+0000","last_became_active":"2024-03-28T18:35:49.931925+0000","last_became_peered":"2024-03-28T18:35:49.931925+0000","last_unstale":"2024-03-28T18:37:22.112826+0000","last_undegraded":"2024-03-28T18:37:22.112826+0000","last_fullsized":"2024-03-28T18:37:22.112826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:26.477117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590784+0000","last_change":"2024-03-28T18:35:50.713654+0000","last_active":"2024-03-28T18:36:54.590784+0000","last_peered":"2024-03-28T18:36:54.590784+0000","last_clean":"2024-03-28T18:36:54.590784+0000","last_became_active":"2024-03-28T18:35:50.713146+0000","last_became_peered":"2024-03-28T18:35:50.713146+0000","last_unstale":"2024-03-28T18:36:54.590784+0000","last_undegraded":"2024-03-28T18:36:54.590784+0000","last_fullsized":"2024-03-28T18:36:54.590784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:23.951963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591861+0000","last_change":"2024-03-28T18:35:55.099331+0000","last_active":"2024-03-28T18:36:54.591861+0000","last_peered":"2024-03-28T18:36:54.591861+0000","last_clean":"2024-03-28T18:36:54.591861+0000","last_became_active":"2024-03-28T18:35:55.098912+0000","last_became_peered":"2024-03-28T18:35:55.098912+0000","last_unstale":"2024-03-28T18:36:54.591861+0000","last_undegraded":"2024-03-28T18:36:54.591861+0000","last_fullsized":"2024-03-28T18:36:54.591861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:31.082148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506783+0000","last_change":"2024-03-28T18:35:50.704377+0000","last_active":"2024-03-28T18:37:23.506783+0000","last_peered":"2024-03-28T18:37:23.506783+0000","last_clean":"2024-03-28T18:37:23.506783+0000","last_became_active":"2024-03-28T18:35:50.703991+0000","last_became_peered":"2024-03-28T18:35:50.703991+0000","last_unstale":"2024-03-28T18:37:23.506783+0000","last_undegraded":"2024-03-28T18:37:23.506783+0000","last_fullsized":"2024-03-28T18:37:23.506783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:48.173466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":104,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:21.094868+0000","last_change":"2024-03-28T18:35:49.933528+0000","last_active":"2024-03-28T18:37:21.094868+0000","last_peered":"2024-03-28T18:37:21.094868+0000","last_clean":"2024-03-28T18:37:21.094868+0000","last_became_active":"2024-03-28T18:35:49.933171+0000","last_became_peered":"2024-03-28T18:35:49.933171+0000","last_unstale":"2024-03-28T18:37:21.094868+0000","last_undegraded":"2024-03-28T18:37:21.094868+0000","last_fullsized":"2024-03-28T18:37:21.094868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:50:59.928400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593337+0000","last_change":"2024-03-28T18:35:55.029752+0000","last_active":"2024-03-28T18:36:54.593337+0000","last_peered":"2024-03-28T18:36:54.593337+0000","last_clean":"2024-03-28T18:36:54.593337+0000","last_became_active":"2024-03-28T18:35:55.029407+0000","last_became_peered":"2024-03-28T18:35:55.029407+0000","last_unstale":"2024-03-28T18:36:54.593337+0000","last_undegraded":"2024-03-28T18:36:54.593337+0000","last_fullsized":"2024-03-28T18:36:54.593337+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:17.328315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591406+0000","last_change":"2024-03-28T18:35:50.706380+0000","last_active":"2024-03-28T18:36:54.591406+0000","last_peered":"2024-03-28T18:36:54.591406+0000","last_clean":"2024-03-28T18:36:54.591406+0000","last_became_active":"2024-03-28T18:35:50.706044+0000","last_became_peered":"2024-03-28T18:35:50.706044+0000","last_unstale":"2024-03-28T18:36:54.591406+0000","last_undegraded":"2024-03-28T18:36:54.591406+0000","last_fullsized":"2024-03-28T18:36:54.591406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:04.916646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589278+0000","last_change":"2024-03-28T18:35:49.930241+0000","last_active":"2024-03-28T18:36:54.589278+0000","last_peered":"2024-03-28T18:36:54.589278+0000","last_clean":"2024-03-28T18:36:54.589278+0000","last_became_active":"2024-03-28T18:35:49.929830+0000","last_became_peered":"2024-03-28T18:35:49.929830+0000","last_unstale":"2024-03-28T18:36:54.589278+0000","last_undegraded":"2024-03-28T18:36:54.589278+0000","last_fullsized":"2024-03-28T18:36:54.589278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:11.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592511+0000","last_change":"2024-03-28T18:34:38.804796+0000","last_active":"2024-03-28T18:36:54.592511+0000","last_peered":"2024-03-28T18:36:54.592511+0000","last_clean":"2024-03-28T18:36:54.592511+0000","last_became_active":"2024-03-28T18:34:38.804419+0000","last_became_peered":"2024-03-28T18:34:38.804419+0000","last_unstale":"2024-03-28T18:36:54.592511+0000","last_undegraded":"2024-03-28T18:36:54.592511+0000","last_fullsized":"2024-03-28T18:36:54.592511+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_clean_scrub_stamp":"2024-03-28T18:31:13.733610+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:38.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589683+0000","last_change":"2024-03-28T18:35:55.099390+0000","last_active":"2024-03-28T18:36:54.589683+0000","last_peered":"2024-03-28T18:36:54.589683+0000","last_clean":"2024-03-28T18:36:54.589683+0000","last_became_active":"2024-03-28T18:35:55.098960+0000","last_became_peered":"2024-03-28T18:35:55.098960+0000","last_unstale":"2024-03-28T18:36:54.589683+0000","last_undegraded":"2024-03-28T18:36:54.589683+0000","last_fullsized":"2024-03-28T18:36:54.589683+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:22.460846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591214+0000","last_change":"2024-03-28T18:35:50.711961+0000","last_active":"2024-03-28T18:36:54.591214+0000","last_peered":"2024-03-28T18:36:54.591214+0000","last_clean":"2024-03-28T18:36:54.591214+0000","last_became_active":"2024-03-28T18:35:50.705466+0000","last_became_peered":"2024-03-28T18:35:50.705466+0000","last_unstale":"2024-03-28T18:36:54.591214+0000","last_undegraded":"2024-03-28T18:36:54.591214+0000","last_fullsized":"2024-03-28T18:36:54.591214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:02.527220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.603568+0000","last_change":"2024-03-28T18:36:54.602798+0000","last_active":"2024-03-28T18:36:54.603568+0000","last_peered":"2024-03-28T18:36:54.603568+0000","last_clean":"2024-03-28T18:36:54.603568+0000","last_became_active":"2024-03-28T18:36:54.599176+0000","last_became_peered":"2024-03-28T18:36:54.599176+0000","last_unstale":"2024-03-28T18:36:54.603568+0000","last_undegraded":"2024-03-28T18:36:54.603568+0000","last_fullsized":"2024-03-28T18:36:54.603568+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:27:17.065687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590414+0000","last_change":"2024-03-28T18:35:55.023402+0000","last_active":"2024-03-28T18:36:54.590414+0000","last_peered":"2024-03-28T18:36:54.590414+0000","last_clean":"2024-03-28T18:36:54.590414+0000","last_became_active":"2024-03-28T18:35:55.023006+0000","last_became_peered":"2024-03-28T18:35:55.023006+0000","last_unstale":"2024-03-28T18:36:54.590414+0000","last_undegraded":"2024-03-28T18:36:54.590414+0000","last_fullsized":"2024-03-28T18:36:54.590414+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:12.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592600+0000","last_change":"2024-03-28T18:35:49.942494+0000","last_active":"2024-03-28T18:36:54.592600+0000","last_peered":"2024-03-28T18:36:54.592600+0000","last_clean":"2024-03-28T18:36:54.592600+0000","last_became_active":"2024-03-28T18:35:49.941607+0000","last_became_peered":"2024-03-28T18:35:49.941607+0000","last_unstale":"2024-03-28T18:36:54.592600+0000","last_undegraded":"2024-03-28T18:36:54.592600+0000","last_fullsized":"2024-03-28T18:36:54.592600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:25.315100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591342+0000","last_change":"2024-03-28T18:35:50.707912+0000","last_active":"2024-03-28T18:36:54.591342+0000","last_peered":"2024-03-28T18:36:54.591342+0000","last_clean":"2024-03-28T18:36:54.591342+0000","last_became_active":"2024-03-28T18:35:50.707304+0000","last_became_peered":"2024-03-28T18:35:50.707304+0000","last_unstale":"2024-03-28T18:36:54.591342+0000","last_undegraded":"2024-03-28T18:36:54.591342+0000","last_fullsized":"2024-03-28T18:36:54.591342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:45.556370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590548+0000","last_change":"2024-03-28T18:35:55.099919+0000","last_active":"2024-03-28T18:36:54.590548+0000","last_peered":"2024-03-28T18:36:54.590548+0000","last_clean":"2024-03-28T18:36:54.590548+0000","last_became_active":"2024-03-28T18:35:55.099430+0000","last_became_peered":"2024-03-28T18:35:55.099430+0000","last_unstale":"2024-03-28T18:36:54.590548+0000","last_undegraded":"2024-03-28T18:36:54.590548+0000","last_fullsized":"2024-03-28T18:36:54.590548+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:30.091473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601614+0000","last_change":"2024-03-28T18:35:49.935871+0000","last_active":"2024-03-28T18:36:54.601614+0000","last_peered":"2024-03-28T18:36:54.601614+0000","last_clean":"2024-03-28T18:36:54.601614+0000","last_became_active":"2024-03-28T18:35:49.935536+0000","last_became_peered":"2024-03-28T18:35:49.935536+0000","last_unstale":"2024-03-28T18:36:54.601614+0000","last_undegraded":"2024-03-28T18:36:54.601614+0000","last_fullsized":"2024-03-28T18:36:54.601614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:22.915067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591720+0000","last_change":"2024-03-28T18:35:50.710877+0000","last_active":"2024-03-28T18:36:54.591720+0000","last_peered":"2024-03-28T18:36:54.591720+0000","last_clean":"2024-03-28T18:36:54.591720+0000","last_became_active":"2024-03-28T18:35:50.709987+0000","last_became_peered":"2024-03-28T18:35:50.709987+0000","last_unstale":"2024-03-28T18:36:54.591720+0000","last_undegraded":"2024-03-28T18:36:54.591720+0000","last_fullsized":"2024-03-28T18:36:54.591720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:43.092541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595346+0000","last_change":"2024-03-28T18:35:55.023390+0000","last_active":"2024-03-28T18:36:54.595346+0000","last_peered":"2024-03-28T18:36:54.595346+0000","last_clean":"2024-03-28T18:36:54.595346+0000","last_became_active":"2024-03-28T18:35:55.023034+0000","last_became_peered":"2024-03-28T18:35:55.023034+0000","last_unstale":"2024-03-28T18:36:54.595346+0000","last_undegraded":"2024-03-28T18:36:54.595346+0000","last_fullsized":"2024-03-28T18:36:54.595346+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:05.829144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590255+0000","last_change":"2024-03-28T18:35:49.932675+0000","last_active":"2024-03-28T18:36:54.590255+0000","last_peered":"2024-03-28T18:36:54.590255+0000","last_clean":"2024-03-28T18:36:54.590255+0000","last_became_active":"2024-03-28T18:35:49.932246+0000","last_became_peered":"2024-03-28T18:35:49.932246+0000","last_unstale":"2024-03-28T18:36:54.590255+0000","last_undegraded":"2024-03-28T18:36:54.590255+0000","last_fullsized":"2024-03-28T18:36:54.590255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:46.806411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594334+0000","last_change":"2024-03-28T18:35:50.716861+0000","last_active":"2024-03-28T18:36:54.594334+0000","last_peered":"2024-03-28T18:36:54.594334+0000","last_clean":"2024-03-28T18:36:54.594334+0000","last_became_active":"2024-03-28T18:35:50.716377+0000","last_became_peered":"2024-03-28T18:35:50.716377+0000","last_unstale":"2024-03-28T18:36:54.594334+0000","last_undegraded":"2024-03-28T18:36:54.594334+0000","last_fullsized":"2024-03-28T18:36:54.594334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:27.018809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589916+0000","last_change":"2024-03-28T18:35:55.099891+0000","last_active":"2024-03-28T18:36:54.589916+0000","last_peered":"2024-03-28T18:36:54.589916+0000","last_clean":"2024-03-28T18:36:54.589916+0000","last_became_active":"2024-03-28T18:35:55.098988+0000","last_became_peered":"2024-03-28T18:35:55.098988+0000","last_unstale":"2024-03-28T18:36:54.589916+0000","last_undegraded":"2024-03-28T18:36:54.589916+0000","last_fullsized":"2024-03-28T18:36:54.589916+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:15.682731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592671+0000","last_change":"2024-03-28T18:35:49.936146+0000","last_active":"2024-03-28T18:36:54.592671+0000","last_peered":"2024-03-28T18:36:54.592671+0000","last_clean":"2024-03-28T18:36:54.592671+0000","last_became_active":"2024-03-28T18:35:49.935771+0000","last_became_peered":"2024-03-28T18:35:49.935771+0000","last_unstale":"2024-03-28T18:36:54.592671+0000","last_undegraded":"2024-03-28T18:36:54.592671+0000","last_fullsized":"2024-03-28T18:36:54.592671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:58.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592025+0000","last_change":"2024-03-28T18:35:50.707527+0000","last_active":"2024-03-28T18:36:54.592025+0000","last_peered":"2024-03-28T18:36:54.592025+0000","last_clean":"2024-03-28T18:36:54.592025+0000","last_became_active":"2024-03-28T18:35:50.707216+0000","last_became_peered":"2024-03-28T18:35:50.707216+0000","last_unstale":"2024-03-28T18:36:54.592025+0000","last_undegraded":"2024-03-28T18:36:54.592025+0000","last_fullsized":"2024-03-28T18:36:54.592025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:32.344898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601648+0000","last_change":"2024-03-28T18:35:55.033294+0000","last_active":"2024-03-28T18:36:54.601648+0000","last_peered":"2024-03-28T18:36:54.601648+0000","last_clean":"2024-03-28T18:36:54.601648+0000","last_became_active":"2024-03-28T18:35:55.030743+0000","last_became_peered":"2024-03-28T18:35:55.030743+0000","last_unstale":"2024-03-28T18:36:54.601648+0000","last_undegraded":"2024-03-28T18:36:54.601648+0000","last_fullsized":"2024-03-28T18:36:54.601648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:16:11.003542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592174+0000","last_change":"2024-03-28T18:35:49.931394+0000","last_active":"2024-03-28T18:36:54.592174+0000","last_peered":"2024-03-28T18:36:54.592174+0000","last_clean":"2024-03-28T18:36:54.592174+0000","last_became_active":"2024-03-28T18:35:49.931045+0000","last_became_peered":"2024-03-28T18:35:49.931045+0000","last_unstale":"2024-03-28T18:36:54.592174+0000","last_undegraded":"2024-03-28T18:36:54.592174+0000","last_fullsized":"2024-03-28T18:36:54.592174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:54.030481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.522605+0000","last_change":"2024-03-28T18:35:50.714745+0000","last_active":"2024-03-28T18:37:23.522605+0000","last_peered":"2024-03-28T18:37:23.522605+0000","last_clean":"2024-03-28T18:37:23.522605+0000","last_became_active":"2024-03-28T18:35:50.712991+0000","last_became_peered":"2024-03-28T18:35:50.712991+0000","last_unstale":"2024-03-28T18:37:23.522605+0000","last_undegraded":"2024-03-28T18:37:23.522605+0000","last_fullsized":"2024-03-28T18:37:23.522605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:34.403662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589455+0000","last_change":"2024-03-28T18:35:55.032353+0000","last_active":"2024-03-28T18:36:54.589455+0000","last_peered":"2024-03-28T18:36:54.589455+0000","last_clean":"2024-03-28T18:36:54.589455+0000","last_became_active":"2024-03-28T18:35:55.031889+0000","last_became_peered":"2024-03-28T18:35:55.031889+0000","last_unstale":"2024-03-28T18:36:54.589455+0000","last_undegraded":"2024-03-28T18:36:54.589455+0000","last_fullsized":"2024-03-28T18:36:54.589455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:56.358941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595619+0000","last_change":"2024-03-28T18:35:49.944107+0000","last_active":"2024-03-28T18:36:54.595619+0000","last_peered":"2024-03-28T18:36:54.595619+0000","last_clean":"2024-03-28T18:36:54.595619+0000","last_became_active":"2024-03-28T18:35:49.943726+0000","last_became_peered":"2024-03-28T18:35:49.943726+0000","last_unstale":"2024-03-28T18:36:54.595619+0000","last_undegraded":"2024-03-28T18:36:54.595619+0000","last_fullsized":"2024-03-28T18:36:54.595619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:05.301302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591044+0000","last_change":"2024-03-28T18:35:50.709236+0000","last_active":"2024-03-28T18:36:54.591044+0000","last_peered":"2024-03-28T18:36:54.591044+0000","last_clean":"2024-03-28T18:36:54.591044+0000","last_became_active":"2024-03-28T18:35:50.708824+0000","last_became_peered":"2024-03-28T18:35:50.708824+0000","last_unstale":"2024-03-28T18:36:54.591044+0000","last_undegraded":"2024-03-28T18:36:54.591044+0000","last_fullsized":"2024-03-28T18:36:54.591044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:21:30.207033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589738+0000","last_change":"2024-03-28T18:35:55.099711+0000","last_active":"2024-03-28T18:36:54.589738+0000","last_peered":"2024-03-28T18:36:54.589738+0000","last_clean":"2024-03-28T18:36:54.589738+0000","last_became_active":"2024-03-28T18:35:55.099054+0000","last_became_peered":"2024-03-28T18:35:55.099054+0000","last_unstale":"2024-03-28T18:36:54.589738+0000","last_undegraded":"2024-03-28T18:36:54.589738+0000","last_fullsized":"2024-03-28T18:36:54.589738+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:22:32.326655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592694+0000","last_change":"2024-03-28T18:35:49.939428+0000","last_active":"2024-03-28T18:36:54.592694+0000","last_peered":"2024-03-28T18:36:54.592694+0000","last_clean":"2024-03-28T18:36:54.592694+0000","last_became_active":"2024-03-28T18:35:49.939089+0000","last_became_peered":"2024-03-28T18:35:49.939089+0000","last_unstale":"2024-03-28T18:36:54.592694+0000","last_undegraded":"2024-03-28T18:36:54.592694+0000","last_fullsized":"2024-03-28T18:36:54.592694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:38.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515324+0000","last_change":"2024-03-28T18:35:50.720931+0000","last_active":"2024-03-28T18:37:23.515324+0000","last_peered":"2024-03-28T18:37:23.515324+0000","last_clean":"2024-03-28T18:37:23.515324+0000","last_became_active":"2024-03-28T18:35:50.720544+0000","last_became_peered":"2024-03-28T18:35:50.720544+0000","last_unstale":"2024-03-28T18:37:23.515324+0000","last_undegraded":"2024-03-28T18:37:23.515324+0000","last_fullsized":"2024-03-28T18:37:23.515324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:05.092688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592432+0000","last_change":"2024-03-28T18:35:55.024518+0000","last_active":"2024-03-28T18:36:54.592432+0000","last_peered":"2024-03-28T18:36:54.592432+0000","last_clean":"2024-03-28T18:36:54.592432+0000","last_became_active":"2024-03-28T18:35:55.024103+0000","last_became_peered":"2024-03-28T18:35:55.024103+0000","last_unstale":"2024-03-28T18:36:54.592432+0000","last_undegraded":"2024-03-28T18:36:54.592432+0000","last_fullsized":"2024-03-28T18:36:54.592432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:32:57.620744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'5","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.625690+0000","last_change":"2024-03-28T18:36:54.602349+0000","last_active":"2024-03-28T18:37:23.625690+0000","last_peered":"2024-03-28T18:37:23.625690+0000","last_clean":"2024-03-28T18:37:23.625690+0000","last_became_active":"2024-03-28T18:36:54.595886+0000","last_became_peered":"2024-03-28T18:36:54.595886+0000","last_unstale":"2024-03-28T18:37:23.625690+0000","last_undegraded":"2024-03-28T18:37:23.625690+0000","last_fullsized":"2024-03-28T18:37:23.625690+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:25:02.437159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.508021+0000","last_change":"2024-03-28T18:35:50.720449+0000","last_active":"2024-03-28T18:37:23.508021+0000","last_peered":"2024-03-28T18:37:23.508021+0000","last_clean":"2024-03-28T18:37:23.508021+0000","last_became_active":"2024-03-28T18:35:50.719840+0000","last_became_peered":"2024-03-28T18:35:50.719840+0000","last_unstale":"2024-03-28T18:37:23.508021+0000","last_undegraded":"2024-03-28T18:37:23.508021+0000","last_fullsized":"2024-03-28T18:37:23.508021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:48.980677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593710+0000","last_change":"2024-03-28T18:35:55.032614+0000","last_active":"2024-03-28T18:36:54.593710+0000","last_peered":"2024-03-28T18:36:54.593710+0000","last_clean":"2024-03-28T18:36:54.593710+0000","last_became_active":"2024-03-28T18:35:55.032180+0000","last_became_peered":"2024-03-28T18:35:55.032180+0000","last_unstale":"2024-03-28T18:36:54.593710+0000","last_undegraded":"2024-03-28T18:36:54.593710+0000","last_fullsized":"2024-03-28T18:36:54.593710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:40.121016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589786+0000","last_change":"2024-03-28T18:35:49.940681+0000","last_active":"2024-03-28T18:36:54.589786+0000","last_peered":"2024-03-28T18:36:54.589786+0000","last_clean":"2024-03-28T18:36:54.589786+0000","last_became_active":"2024-03-28T18:35:49.940376+0000","last_became_peered":"2024-03-28T18:35:49.940376+0000","last_unstale":"2024-03-28T18:36:54.589786+0000","last_undegraded":"2024-03-28T18:36:54.589786+0000","last_fullsized":"2024-03-28T18:36:54.589786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:02:11.926285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506994+0000","last_change":"2024-03-28T18:35:50.715051+0000","last_active":"2024-03-28T18:37:23.506994+0000","last_peered":"2024-03-28T18:37:23.506994+0000","last_clean":"2024-03-28T18:37:23.506994+0000","last_became_active":"2024-03-28T18:35:50.714647+0000","last_became_peered":"2024-03-28T18:35:50.714647+0000","last_unstale":"2024-03-28T18:37:23.506994+0000","last_undegraded":"2024-03-28T18:37:23.506994+0000","last_fullsized":"2024-03-28T18:37:23.506994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:02.832665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588600+0000","last_change":"2024-03-28T18:35:55.021869+0000","last_active":"2024-03-28T18:36:54.588600+0000","last_peered":"2024-03-28T18:36:54.588600+0000","last_clean":"2024-03-28T18:36:54.588600+0000","last_became_active":"2024-03-28T18:35:55.021535+0000","last_became_peered":"2024-03-28T18:35:55.021535+0000","last_unstale":"2024-03-28T18:36:54.588600+0000","last_undegraded":"2024-03-28T18:36:54.588600+0000","last_fullsized":"2024-03-28T18:36:54.588600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:31.610882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591586+0000","last_change":"2024-03-28T18:35:49.936095+0000","last_active":"2024-03-28T18:36:54.591586+0000","last_peered":"2024-03-28T18:36:54.591586+0000","last_clean":"2024-03-28T18:36:54.591586+0000","last_became_active":"2024-03-28T18:35:49.935766+0000","last_became_peered":"2024-03-28T18:35:49.935766+0000","last_unstale":"2024-03-28T18:36:54.591586+0000","last_undegraded":"2024-03-28T18:36:54.591586+0000","last_fullsized":"2024-03-28T18:36:54.591586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:00:06.712979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.507860+0000","last_change":"2024-03-28T18:35:50.715313+0000","last_active":"2024-03-28T18:37:23.507860+0000","last_peered":"2024-03-28T18:37:23.507860+0000","last_clean":"2024-03-28T18:37:23.507860+0000","last_became_active":"2024-03-28T18:35:50.714923+0000","last_became_peered":"2024-03-28T18:35:50.714923+0000","last_unstale":"2024-03-28T18:37:23.507860+0000","last_undegraded":"2024-03-28T18:37:23.507860+0000","last_fullsized":"2024-03-28T18:37:23.507860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:49.406551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.504412+0000","last_change":"2024-03-28T18:35:50.714244+0000","last_active":"2024-03-28T18:37:23.504412+0000","last_peered":"2024-03-28T18:37:23.504412+0000","last_clean":"2024-03-28T18:37:23.504412+0000","last_became_active":"2024-03-28T18:35:50.713347+0000","last_became_peered":"2024-03-28T18:35:50.713347+0000","last_unstale":"2024-03-28T18:37:23.504412+0000","last_undegraded":"2024-03-28T18:37:23.504412+0000","last_fullsized":"2024-03-28T18:37:23.504412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:34.360341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.505208+0000","last_change":"2024-03-28T18:35:50.711132+0000","last_active":"2024-03-28T18:37:23.505208+0000","last_peered":"2024-03-28T18:37:23.505208+0000","last_clean":"2024-03-28T18:37:23.505208+0000","last_became_active":"2024-03-28T18:35:50.710500+0000","last_became_peered":"2024-03-28T18:35:50.710500+0000","last_unstale":"2024-03-28T18:37:23.505208+0000","last_undegraded":"2024-03-28T18:37:23.505208+0000","last_fullsized":"2024-03-28T18:37:23.505208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:22.060106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.529114+0000","last_change":"2024-03-28T18:35:50.712396+0000","last_active":"2024-03-28T18:37:23.529114+0000","last_peered":"2024-03-28T18:37:23.529114+0000","last_clean":"2024-03-28T18:37:23.529114+0000","last_became_active":"2024-03-28T18:35:50.711950+0000","last_became_peered":"2024-03-28T18:35:50.711950+0000","last_unstale":"2024-03-28T18:37:23.529114+0000","last_undegraded":"2024-03-28T18:37:23.529114+0000","last_fullsized":"2024-03-28T18:37:23.529114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:24.573754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593897+0000","last_change":"2024-03-28T18:35:50.718222+0000","last_active":"2024-03-28T18:36:54.593897+0000","last_peered":"2024-03-28T18:36:54.593897+0000","last_clean":"2024-03-28T18:36:54.593897+0000","last_became_active":"2024-03-28T18:35:50.717234+0000","last_became_peered":"2024-03-28T18:35:50.717234+0000","last_unstale":"2024-03-28T18:36:54.593897+0000","last_undegraded":"2024-03-28T18:36:54.593897+0000","last_fullsized":"2024-03-28T18:36:54.593897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:28:47.544280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591257+0000","last_change":"2024-03-28T18:35:50.707486+0000","last_active":"2024-03-28T18:36:54.591257+0000","last_peered":"2024-03-28T18:36:54.591257+0000","last_clean":"2024-03-28T18:36:54.591257+0000","last_became_active":"2024-03-28T18:35:50.706902+0000","last_became_peered":"2024-03-28T18:35:50.706902+0000","last_unstale":"2024-03-28T18:36:54.591257+0000","last_undegraded":"2024-03-28T18:36:54.591257+0000","last_fullsized":"2024-03-28T18:36:54.591257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:00.160873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503369+0000","last_change":"2024-03-28T18:35:50.715566+0000","last_active":"2024-03-28T18:37:23.503369+0000","last_peered":"2024-03-28T18:37:23.503369+0000","last_clean":"2024-03-28T18:37:23.503369+0000","last_became_active":"2024-03-28T18:35:50.715193+0000","last_became_peered":"2024-03-28T18:35:50.715193+0000","last_unstale":"2024-03-28T18:37:23.503369+0000","last_undegraded":"2024-03-28T18:37:23.503369+0000","last_fullsized":"2024-03-28T18:37:23.503369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:57.126250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514038+0000","last_change":"2024-03-28T18:35:50.709904+0000","last_active":"2024-03-28T18:37:23.514038+0000","last_peered":"2024-03-28T18:37:23.514038+0000","last_clean":"2024-03-28T18:37:23.514038+0000","last_became_active":"2024-03-28T18:35:50.709575+0000","last_became_peered":"2024-03-28T18:35:50.709575+0000","last_unstale":"2024-03-28T18:37:23.514038+0000","last_undegraded":"2024-03-28T18:37:23.514038+0000","last_fullsized":"2024-03-28T18:37:23.514038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:26.580718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.598585+0000","last_change":"2024-03-28T18:36:54.598585+0000","last_active":"2024-03-28T18:36:54.598585+0000","last_peered":"2024-03-28T18:36:54.598585+0000","last_clean":"2024-03-28T18:36:54.598585+0000","last_became_active":"2024-03-28T18:36:54.598262+0000","last_became_peered":"2024-03-28T18:36:54.598262+0000","last_unstale":"2024-03-28T18:36:54.598585+0000","last_undegraded":"2024-03-28T18:36:54.598585+0000","last_fullsized":"2024-03-28T18:36:54.598585+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:46.028970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521405+0000","last_change":"2024-03-28T18:35:50.707563+0000","last_active":"2024-03-28T18:37:23.521405+0000","last_peered":"2024-03-28T18:37:23.521405+0000","last_clean":"2024-03-28T18:37:23.521405+0000","last_became_active":"2024-03-28T18:35:50.706300+0000","last_became_peered":"2024-03-28T18:35:50.706300+0000","last_unstale":"2024-03-28T18:37:23.521405+0000","last_undegraded":"2024-03-28T18:37:23.521405+0000","last_fullsized":"2024-03-28T18:37:23.521405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:05.324365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592344+0000","last_change":"2024-03-28T18:35:50.705549+0000","last_active":"2024-03-28T18:36:54.592344+0000","last_peered":"2024-03-28T18:36:54.592344+0000","last_clean":"2024-03-28T18:36:54.592344+0000","last_became_active":"2024-03-28T18:35:50.705190+0000","last_became_peered":"2024-03-28T18:35:50.705190+0000","last_unstale":"2024-03-28T18:36:54.592344+0000","last_undegraded":"2024-03-28T18:36:54.592344+0000","last_fullsized":"2024-03-28T18:36:54.592344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:38.662227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521023+0000","last_change":"2024-03-28T18:35:50.719591+0000","last_active":"2024-03-28T18:37:23.521023+0000","last_peered":"2024-03-28T18:37:23.521023+0000","last_clean":"2024-03-28T18:37:23.521023+0000","last_became_active":"2024-03-28T18:35:50.719272+0000","last_became_peered":"2024-03-28T18:35:50.719272+0000","last_unstale":"2024-03-28T18:37:23.521023+0000","last_undegraded":"2024-03-28T18:37:23.521023+0000","last_fullsized":"2024-03-28T18:37:23.521023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:06.627761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592065+0000","last_change":"2024-03-28T18:35:50.712965+0000","last_active":"2024-03-28T18:36:54.592065+0000","last_peered":"2024-03-28T18:36:54.592065+0000","last_clean":"2024-03-28T18:36:54.592065+0000","last_became_active":"2024-03-28T18:35:50.712648+0000","last_became_peered":"2024-03-28T18:35:50.712648+0000","last_unstale":"2024-03-28T18:36:54.592065+0000","last_undegraded":"2024-03-28T18:36:54.592065+0000","last_fullsized":"2024-03-28T18:36:54.592065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:00.925664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506395+0000","last_change":"2024-03-28T18:35:50.717277+0000","last_active":"2024-03-28T18:37:23.506395+0000","last_peered":"2024-03-28T18:37:23.506395+0000","last_clean":"2024-03-28T18:37:23.506395+0000","last_became_active":"2024-03-28T18:35:50.716161+0000","last_became_peered":"2024-03-28T18:35:50.716161+0000","last_unstale":"2024-03-28T18:37:23.506395+0000","last_undegraded":"2024-03-28T18:37:23.506395+0000","last_fullsized":"2024-03-28T18:37:23.506395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:42.789575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210208,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68105,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":699,"num_read_kb":440,"num_write":243,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1282,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":204672,"data_compressed":4791,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318304,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":511389,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32328,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712824,"statfs":{"total":95995035648,"available":95961931776,"internally_reserved":0,"allocated":1171456,"data_stored":1159635,"data_compressed":57783,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743467,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":515303,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005103,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":511473,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266741,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":1108223,"data_compressed":56186,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":506711,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":570311,"data_compressed":22201,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":508167,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":567942,"data_compressed":22201,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":506419,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":511176,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724606,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":1103273,"data_compressed":56186,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64322,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62202,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63278,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:37:27.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:37:28.114 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:28 smithi145 ceph-mon[32151]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:28.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:28 smithi145 ceph-mon[32151]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:28 smithi192 ceph-mon[36589]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:28 smithi192 ceph-mon[36589]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:28.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:28 smithi164 ceph-mon[36248]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:28 smithi164 ceph-mon[36248]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:29.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 -- 172.21.15.145:0/1304110331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103b60 msgr2=0x7f552810c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:29.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 --2- 172.21.15.145:0/1304110331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103b60 0x7f552810c050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f551c009990 tx=0x7f551c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.909 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 -- 172.21.15.145:0/1304110331 shutdown_connections 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 --2- 172.21.15.145:0/1304110331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 0x7f552810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 --2- 172.21.15.145:0/1304110331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103240 0x7f5528103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 --2- 172.21.15.145:0/1304110331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103b60 0x7f552810c050 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 -- 172.21.15.145:0/1304110331 >> 172.21.15.145:0/1304110331 conn(0x7f55280fca50 msgr2=0x7f55280fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 -- 172.21.15.145:0/1304110331 shutdown_connections 2024-03-28T18:37:29.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.908+0000 7f552759e700 1 -- 172.21.15.145:0/1304110331 wait complete. 2024-03-28T18:37:29.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 Processor -- start 2024-03-28T18:37:29.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 -- start start 2024-03-28T18:37:29.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:29.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103b60 0x7f552819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 0x7f55281986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528112390 con 0x7f5528103240 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528112210 con 0x7f5528103b60 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552759e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528112510 con 0x7f552810c670 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:47442/0 (socket says 172.21.15.145:47442) 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 -- 172.21.15.145:0/1119843768 learned_addr learned my addr 172.21.15.145:0/1119843768 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f5525d9b700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103b60 0x7f552819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f5526d9d700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 0x7f55281986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 -- 172.21.15.145:0/1119843768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 msgr2=0x7f55281986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 0x7f55281986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.913 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 -- 172.21.15.145:0/1119843768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103b60 msgr2=0x7f552819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103b60 0x7f552819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.909+0000 7f552659c700 1 -- 172.21.15.145:0/1119843768 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5528198f90 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f552659c700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f551800eb30 tx=0x7f551800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551800cbd0 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5528199280 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5518004500 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518010430 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.910+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55281a5560 con 0x7f5528103240 2024-03-28T18:37:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.911+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552806afb0 con 0x7f5528103240 2024-03-28T18:37:29.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.912+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5518003f00 con 0x7f5528103240 2024-03-28T18:37:29.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.912+0000 7f55177fe700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 0x7f55100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:29.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.913+0000 7f5525d9b700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 0x7f55100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:29.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.913+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f55180cf250 con 0x7f5528103240 2024-03-28T18:37:29.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.913+0000 7f5525d9b700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 0x7f55100a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f551c006010 tx=0x7f551c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:29.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:29.916+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f551809bfc0 con 0x7f5528103240 2024-03-28T18:37:30.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.153+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5528199be0 con 0x7f55100a2890 2024-03-28T18:37:30.163 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.162+0000 7f55177fe700 1 -- 172.21.15.145:0/1119843768 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340891 (secure 0 0 0) 0x7f5528199be0 con 0x7f55100a2890 2024-03-28T18:37:30.167 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.165+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 msgr2=0x7f55100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.165+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 0x7f55100a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f551c006010 tx=0x7f551c005960 comp rx=0 tx=0).stop 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.165+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 msgr2=0x7f552819e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.165+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f551800eb30 tx=0x7f551800ee40 comp rx=0 tx=0).stop 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 shutdown_connections 2024-03-28T18:37:30.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f55100a2890 0x7f55100a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f552810c670 0x7f55281986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5528103240 0x7f552819e5e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 --2- 172.21.15.145:0/1119843768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5528103b60 0x7f552819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 >> 172.21.15.145:0/1119843768 conn(0x7f55280fca50 msgr2=0x7f55280fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 shutdown_connections 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:30.166+0000 7f552759e700 1 -- 172.21.15.145:0/1119843768 wait complete. 2024-03-28T18:37:30.171 INFO:teuthology.orchestra.run.smithi145.stderr:dumped all 2024-03-28T18:37:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:30 smithi192 ceph-mon[36589]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:37:30.961 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:30 smithi145 ceph-mon[32151]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:37:30.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:30 smithi164 ceph-mon[36248]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:37:30.987 INFO:teuthology.orchestra.run.smithi145.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-03-28T18:37:29.498711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658956,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":867,"num_read_kb":571,"num_write":430,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182018,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370044,"kb_used_data":12572,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571780,"statfs":{"total":1151940427776,"available":1151561502720,"internally_reserved":0,"allocated":12873728,"data_stored":8080802,"data_compressed":358785,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":57,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002890"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.523470+0000","last_change":"2024-03-28T18:35:50.713920+0000","last_active":"2024-03-28T18:37:23.523470+0000","last_peered":"2024-03-28T18:37:23.523470+0000","last_clean":"2024-03-28T18:37:23.523470+0000","last_became_active":"2024-03-28T18:35:50.712723+0000","last_became_peered":"2024-03-28T18:35:50.712723+0000","last_unstale":"2024-03-28T18:37:23.523470+0000","last_undegraded":"2024-03-28T18:37:23.523470+0000","last_fullsized":"2024-03-28T18:37:23.523470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:31.562072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.530931+0000","last_change":"2024-03-28T18:35:50.714271+0000","last_active":"2024-03-28T18:37:23.530931+0000","last_peered":"2024-03-28T18:37:23.530931+0000","last_clean":"2024-03-28T18:37:23.530931+0000","last_became_active":"2024-03-28T18:35:50.713461+0000","last_became_peered":"2024-03-28T18:35:50.713461+0000","last_unstale":"2024-03-28T18:37:23.530931+0000","last_undegraded":"2024-03-28T18:37:23.530931+0000","last_fullsized":"2024-03-28T18:37:23.530931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:03:47.789033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.532217+0000","last_change":"2024-03-28T18:35:50.720511+0000","last_active":"2024-03-28T18:37:23.532217+0000","last_peered":"2024-03-28T18:37:23.532217+0000","last_clean":"2024-03-28T18:37:23.532217+0000","last_became_active":"2024-03-28T18:35:50.720227+0000","last_became_peered":"2024-03-28T18:35:50.720227+0000","last_unstale":"2024-03-28T18:37:23.532217+0000","last_undegraded":"2024-03-28T18:37:23.532217+0000","last_fullsized":"2024-03-28T18:37:23.532217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:19.849187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.489081+0000","last_change":"2024-03-28T18:35:50.707852+0000","last_active":"2024-03-28T18:37:23.489081+0000","last_peered":"2024-03-28T18:37:23.489081+0000","last_clean":"2024-03-28T18:37:23.489081+0000","last_became_active":"2024-03-28T18:35:50.705465+0000","last_became_peered":"2024-03-28T18:35:50.705465+0000","last_unstale":"2024-03-28T18:37:23.489081+0000","last_undegraded":"2024-03-28T18:37:23.489081+0000","last_fullsized":"2024-03-28T18:37:23.489081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:00.251387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521714+0000","last_change":"2024-03-28T18:35:50.716142+0000","last_active":"2024-03-28T18:37:23.521714+0000","last_peered":"2024-03-28T18:37:23.521714+0000","last_clean":"2024-03-28T18:37:23.521714+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:37:23.521714+0000","last_undegraded":"2024-03-28T18:37:23.521714+0000","last_fullsized":"2024-03-28T18:37:23.521714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:04:46.515184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515036+0000","last_change":"2024-03-28T18:36:54.602406+0000","last_active":"2024-03-28T18:37:23.515036+0000","last_peered":"2024-03-28T18:37:23.515036+0000","last_clean":"2024-03-28T18:37:23.515036+0000","last_became_active":"2024-03-28T18:36:54.602016+0000","last_became_peered":"2024-03-28T18:36:54.602016+0000","last_unstale":"2024-03-28T18:37:23.515036+0000","last_undegraded":"2024-03-28T18:37:23.515036+0000","last_fullsized":"2024-03-28T18:37:23.515036+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:24:21.352607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.501388+0000","last_change":"2024-03-28T18:35:50.713781+0000","last_active":"2024-03-28T18:37:23.501388+0000","last_peered":"2024-03-28T18:37:23.501388+0000","last_clean":"2024-03-28T18:37:23.501388+0000","last_became_active":"2024-03-28T18:35:50.713400+0000","last_became_peered":"2024-03-28T18:35:50.713400+0000","last_unstale":"2024-03-28T18:37:23.501388+0000","last_undegraded":"2024-03-28T18:37:23.501388+0000","last_fullsized":"2024-03-28T18:37:23.501388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:37:21.368701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.524907+0000","last_change":"2024-03-28T18:35:50.704912+0000","last_active":"2024-03-28T18:37:23.524907+0000","last_peered":"2024-03-28T18:37:23.524907+0000","last_clean":"2024-03-28T18:37:23.524907+0000","last_became_active":"2024-03-28T18:35:50.704590+0000","last_became_peered":"2024-03-28T18:35:50.704590+0000","last_unstale":"2024-03-28T18:37:23.524907+0000","last_undegraded":"2024-03-28T18:37:23.524907+0000","last_fullsized":"2024-03-28T18:37:23.524907+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:05:17.222377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.510794+0000","last_change":"2024-03-28T18:35:50.710448+0000","last_active":"2024-03-28T18:37:23.510794+0000","last_peered":"2024-03-28T18:37:23.510794+0000","last_clean":"2024-03-28T18:37:23.510794+0000","last_became_active":"2024-03-28T18:35:50.708630+0000","last_became_peered":"2024-03-28T18:35:50.708630+0000","last_unstale":"2024-03-28T18:37:23.510794+0000","last_undegraded":"2024-03-28T18:37:23.510794+0000","last_fullsized":"2024-03-28T18:37:23.510794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:01.711677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.494631+0000","last_change":"2024-03-28T18:35:50.711851+0000","last_active":"2024-03-28T18:37:23.494631+0000","last_peered":"2024-03-28T18:37:23.494631+0000","last_clean":"2024-03-28T18:37:23.494631+0000","last_became_active":"2024-03-28T18:35:50.711497+0000","last_became_peered":"2024-03-28T18:35:50.711497+0000","last_unstale":"2024-03-28T18:37:23.494631+0000","last_undegraded":"2024-03-28T18:37:23.494631+0000","last_fullsized":"2024-03-28T18:37:23.494631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:32.032471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503776+0000","last_change":"2024-03-28T18:35:50.722741+0000","last_active":"2024-03-28T18:37:23.503776+0000","last_peered":"2024-03-28T18:37:23.503776+0000","last_clean":"2024-03-28T18:37:23.503776+0000","last_became_active":"2024-03-28T18:35:50.722367+0000","last_became_peered":"2024-03-28T18:35:50.722367+0000","last_unstale":"2024-03-28T18:37:23.503776+0000","last_undegraded":"2024-03-28T18:37:23.503776+0000","last_fullsized":"2024-03-28T18:37:23.503776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:28:28.898030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.489866+0000","last_change":"2024-03-28T18:35:50.706167+0000","last_active":"2024-03-28T18:37:23.489866+0000","last_peered":"2024-03-28T18:37:23.489866+0000","last_clean":"2024-03-28T18:37:23.489866+0000","last_became_active":"2024-03-28T18:35:50.705485+0000","last_became_peered":"2024-03-28T18:35:50.705485+0000","last_unstale":"2024-03-28T18:37:23.489866+0000","last_undegraded":"2024-03-28T18:37:23.489866+0000","last_fullsized":"2024-03-28T18:37:23.489866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:44:57.492095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515111+0000","last_change":"2024-03-28T18:35:50.716920+0000","last_active":"2024-03-28T18:37:23.515111+0000","last_peered":"2024-03-28T18:37:23.515111+0000","last_clean":"2024-03-28T18:37:23.515111+0000","last_became_active":"2024-03-28T18:35:50.712942+0000","last_became_peered":"2024-03-28T18:35:50.712942+0000","last_unstale":"2024-03-28T18:37:23.515111+0000","last_undegraded":"2024-03-28T18:37:23.515111+0000","last_fullsized":"2024-03-28T18:37:23.515111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:22.550943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.530862+0000","last_change":"2024-03-28T18:35:50.715257+0000","last_active":"2024-03-28T18:37:23.530862+0000","last_peered":"2024-03-28T18:37:23.530862+0000","last_clean":"2024-03-28T18:37:23.530862+0000","last_became_active":"2024-03-28T18:35:50.714943+0000","last_became_peered":"2024-03-28T18:35:50.714943+0000","last_unstale":"2024-03-28T18:37:23.530862+0000","last_undegraded":"2024-03-28T18:37:23.530862+0000","last_fullsized":"2024-03-28T18:37:23.530862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:39:05.683273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.532504+0000","last_change":"2024-03-28T18:35:50.718042+0000","last_active":"2024-03-28T18:37:23.532504+0000","last_peered":"2024-03-28T18:37:23.532504+0000","last_clean":"2024-03-28T18:37:23.532504+0000","last_became_active":"2024-03-28T18:35:50.717713+0000","last_became_peered":"2024-03-28T18:35:50.717713+0000","last_unstale":"2024-03-28T18:37:23.532504+0000","last_undegraded":"2024-03-28T18:37:23.532504+0000","last_fullsized":"2024-03-28T18:37:23.532504+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:08:51.764882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.531651+0000","last_change":"2024-03-28T18:35:50.720741+0000","last_active":"2024-03-28T18:37:23.531651+0000","last_peered":"2024-03-28T18:37:23.531651+0000","last_clean":"2024-03-28T18:37:23.531651+0000","last_became_active":"2024-03-28T18:35:50.720038+0000","last_became_peered":"2024-03-28T18:35:50.720038+0000","last_unstale":"2024-03-28T18:37:23.531651+0000","last_undegraded":"2024-03-28T18:37:23.531651+0000","last_fullsized":"2024-03-28T18:37:23.531651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:49:48.001768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.523700+0000","last_change":"2024-03-28T18:35:50.714152+0000","last_active":"2024-03-28T18:37:23.523700+0000","last_peered":"2024-03-28T18:37:23.523700+0000","last_clean":"2024-03-28T18:37:23.523700+0000","last_became_active":"2024-03-28T18:35:50.713813+0000","last_became_peered":"2024-03-28T18:35:50.713813+0000","last_unstale":"2024-03-28T18:37:23.523700+0000","last_undegraded":"2024-03-28T18:37:23.523700+0000","last_fullsized":"2024-03-28T18:37:23.523700+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:48.698925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.512794+0000","last_change":"2024-03-28T18:35:50.708046+0000","last_active":"2024-03-28T18:37:23.512794+0000","last_peered":"2024-03-28T18:37:23.512794+0000","last_clean":"2024-03-28T18:37:23.512794+0000","last_became_active":"2024-03-28T18:35:50.707694+0000","last_became_peered":"2024-03-28T18:35:50.707694+0000","last_unstale":"2024-03-28T18:37:23.512794+0000","last_undegraded":"2024-03-28T18:37:23.512794+0000","last_fullsized":"2024-03-28T18:37:23.512794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:37:46.494020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514980+0000","last_change":"2024-03-28T18:35:50.714366+0000","last_active":"2024-03-28T18:37:23.514980+0000","last_peered":"2024-03-28T18:37:23.514980+0000","last_clean":"2024-03-28T18:37:23.514980+0000","last_became_active":"2024-03-28T18:35:50.714046+0000","last_became_peered":"2024-03-28T18:35:50.714046+0000","last_unstale":"2024-03-28T18:37:23.514980+0000","last_undegraded":"2024-03-28T18:37:23.514980+0000","last_fullsized":"2024-03-28T18:37:23.514980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:24.757304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595794+0000","last_change":"2024-03-28T18:35:55.032307+0000","last_active":"2024-03-28T18:36:54.595794+0000","last_peered":"2024-03-28T18:36:54.595794+0000","last_clean":"2024-03-28T18:36:54.595794+0000","last_became_active":"2024-03-28T18:35:55.031787+0000","last_became_peered":"2024-03-28T18:35:55.031787+0000","last_unstale":"2024-03-28T18:36:54.595794+0000","last_undegraded":"2024-03-28T18:36:54.595794+0000","last_fullsized":"2024-03-28T18:36:54.595794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:32:15.649053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":113,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.113327+0000","last_change":"2024-03-28T18:35:49.933942+0000","last_active":"2024-03-28T18:37:23.113327+0000","last_peered":"2024-03-28T18:37:23.113327+0000","last_clean":"2024-03-28T18:37:23.113327+0000","last_became_active":"2024-03-28T18:35:49.933608+0000","last_became_peered":"2024-03-28T18:35:49.933608+0000","last_unstale":"2024-03-28T18:37:23.113327+0000","last_undegraded":"2024-03-28T18:37:23.113327+0000","last_fullsized":"2024-03-28T18:37:23.113327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:33:24.327806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.519674+0000","last_change":"2024-03-28T18:35:50.704543+0000","last_active":"2024-03-28T18:37:23.519674+0000","last_peered":"2024-03-28T18:37:23.519674+0000","last_clean":"2024-03-28T18:37:23.519674+0000","last_became_active":"2024-03-28T18:35:50.704232+0000","last_became_peered":"2024-03-28T18:35:50.704232+0000","last_unstale":"2024-03-28T18:37:23.519674+0000","last_undegraded":"2024-03-28T18:37:23.519674+0000","last_fullsized":"2024-03-28T18:37:23.519674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:52:30.934620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589094+0000","last_change":"2024-03-28T18:35:55.032545+0000","last_active":"2024-03-28T18:36:54.589094+0000","last_peered":"2024-03-28T18:36:54.589094+0000","last_clean":"2024-03-28T18:36:54.589094+0000","last_became_active":"2024-03-28T18:35:55.032011+0000","last_became_peered":"2024-03-28T18:35:55.032011+0000","last_unstale":"2024-03-28T18:36:54.589094+0000","last_undegraded":"2024-03-28T18:36:54.589094+0000","last_fullsized":"2024-03-28T18:36:54.589094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:22:46.960816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591842+0000","last_change":"2024-03-28T18:35:49.940627+0000","last_active":"2024-03-28T18:36:54.591842+0000","last_peered":"2024-03-28T18:36:54.591842+0000","last_clean":"2024-03-28T18:36:54.591842+0000","last_became_active":"2024-03-28T18:35:49.940313+0000","last_became_peered":"2024-03-28T18:35:49.940313+0000","last_unstale":"2024-03-28T18:36:54.591842+0000","last_undegraded":"2024-03-28T18:36:54.591842+0000","last_fullsized":"2024-03-28T18:36:54.591842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:24.215713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.530899+0000","last_change":"2024-03-28T18:35:50.720053+0000","last_active":"2024-03-28T18:37:23.530899+0000","last_peered":"2024-03-28T18:37:23.530899+0000","last_clean":"2024-03-28T18:37:23.530899+0000","last_became_active":"2024-03-28T18:35:50.719698+0000","last_became_peered":"2024-03-28T18:35:50.719698+0000","last_unstale":"2024-03-28T18:37:23.530899+0000","last_undegraded":"2024-03-28T18:37:23.530899+0000","last_fullsized":"2024-03-28T18:37:23.530899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:30.758840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.598163+0000","last_change":"2024-03-28T18:36:54.598163+0000","last_active":"2024-03-28T18:36:54.598163+0000","last_peered":"2024-03-28T18:36:54.598163+0000","last_clean":"2024-03-28T18:36:54.598163+0000","last_became_active":"2024-03-28T18:36:54.597729+0000","last_became_peered":"2024-03-28T18:36:54.597729+0000","last_unstale":"2024-03-28T18:36:54.598163+0000","last_undegraded":"2024-03-28T18:36:54.598163+0000","last_fullsized":"2024-03-28T18:36:54.598163+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:27:04.471576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592704+0000","last_change":"2024-03-28T18:35:49.938447+0000","last_active":"2024-03-28T18:36:54.592704+0000","last_peered":"2024-03-28T18:36:54.592704+0000","last_clean":"2024-03-28T18:36:54.592704+0000","last_became_active":"2024-03-28T18:35:49.938125+0000","last_became_peered":"2024-03-28T18:35:49.938125+0000","last_unstale":"2024-03-28T18:36:54.592704+0000","last_undegraded":"2024-03-28T18:36:54.592704+0000","last_fullsized":"2024-03-28T18:36:54.592704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:06:31.064360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515806+0000","last_change":"2024-03-28T18:35:50.711208+0000","last_active":"2024-03-28T18:37:23.515806+0000","last_peered":"2024-03-28T18:37:23.515806+0000","last_clean":"2024-03-28T18:37:23.515806+0000","last_became_active":"2024-03-28T18:35:50.710306+0000","last_became_peered":"2024-03-28T18:35:50.710306+0000","last_unstale":"2024-03-28T18:37:23.515806+0000","last_undegraded":"2024-03-28T18:37:23.515806+0000","last_fullsized":"2024-03-28T18:37:23.515806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.449619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589142+0000","last_change":"2024-03-28T18:35:55.027968+0000","last_active":"2024-03-28T18:36:54.589142+0000","last_peered":"2024-03-28T18:36:54.589142+0000","last_clean":"2024-03-28T18:36:54.589142+0000","last_became_active":"2024-03-28T18:35:55.027486+0000","last_became_peered":"2024-03-28T18:35:55.027486+0000","last_unstale":"2024-03-28T18:36:54.589142+0000","last_undegraded":"2024-03-28T18:36:54.589142+0000","last_fullsized":"2024-03-28T18:36:54.589142+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:49:30.088482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592370+0000","last_change":"2024-03-28T18:35:49.936488+0000","last_active":"2024-03-28T18:36:54.592370+0000","last_peered":"2024-03-28T18:36:54.592370+0000","last_clean":"2024-03-28T18:36:54.592370+0000","last_became_active":"2024-03-28T18:35:49.936122+0000","last_became_peered":"2024-03-28T18:35:49.936122+0000","last_unstale":"2024-03-28T18:36:54.592370+0000","last_undegraded":"2024-03-28T18:36:54.592370+0000","last_fullsized":"2024-03-28T18:36:54.592370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:49:51.096079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.511466+0000","last_change":"2024-03-28T18:35:50.704075+0000","last_active":"2024-03-28T18:37:23.511466+0000","last_peered":"2024-03-28T18:37:23.511466+0000","last_clean":"2024-03-28T18:37:23.511466+0000","last_became_active":"2024-03-28T18:35:50.703726+0000","last_became_peered":"2024-03-28T18:35:50.703726+0000","last_unstale":"2024-03-28T18:37:23.511466+0000","last_undegraded":"2024-03-28T18:37:23.511466+0000","last_fullsized":"2024-03-28T18:37:23.511466+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:08.522098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596528+0000","last_change":"2024-03-28T18:35:55.023382+0000","last_active":"2024-03-28T18:36:54.596528+0000","last_peered":"2024-03-28T18:36:54.596528+0000","last_clean":"2024-03-28T18:36:54.596528+0000","last_became_active":"2024-03-28T18:35:55.022992+0000","last_became_peered":"2024-03-28T18:35:55.022992+0000","last_unstale":"2024-03-28T18:36:54.596528+0000","last_undegraded":"2024-03-28T18:36:54.596528+0000","last_fullsized":"2024-03-28T18:36:54.596528+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:09.260173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":111,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:27.091738+0000","last_change":"2024-03-28T18:35:49.939304+0000","last_active":"2024-03-28T18:37:27.091738+0000","last_peered":"2024-03-28T18:37:27.091738+0000","last_clean":"2024-03-28T18:37:27.091738+0000","last_became_active":"2024-03-28T18:35:49.938976+0000","last_became_peered":"2024-03-28T18:35:49.938976+0000","last_unstale":"2024-03-28T18:37:27.091738+0000","last_undegraded":"2024-03-28T18:37:27.091738+0000","last_fullsized":"2024-03-28T18:37:27.091738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:37.368257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.494281+0000","last_change":"2024-03-28T18:35:50.715841+0000","last_active":"2024-03-28T18:37:23.494281+0000","last_peered":"2024-03-28T18:37:23.494281+0000","last_clean":"2024-03-28T18:37:23.494281+0000","last_became_active":"2024-03-28T18:35:50.715365+0000","last_became_peered":"2024-03-28T18:35:50.715365+0000","last_unstale":"2024-03-28T18:37:23.494281+0000","last_undegraded":"2024-03-28T18:37:23.494281+0000","last_fullsized":"2024-03-28T18:37:23.494281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:33.009378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596592+0000","last_change":"2024-03-28T18:35:55.033073+0000","last_active":"2024-03-28T18:36:54.596592+0000","last_peered":"2024-03-28T18:36:54.596592+0000","last_clean":"2024-03-28T18:36:54.596592+0000","last_became_active":"2024-03-28T18:35:55.032649+0000","last_became_peered":"2024-03-28T18:35:55.032649+0000","last_unstale":"2024-03-28T18:36:54.596592+0000","last_undegraded":"2024-03-28T18:36:54.596592+0000","last_fullsized":"2024-03-28T18:36:54.596592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:14.915178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591110+0000","last_change":"2024-03-28T18:35:49.932672+0000","last_active":"2024-03-28T18:36:54.591110+0000","last_peered":"2024-03-28T18:36:54.591110+0000","last_clean":"2024-03-28T18:36:54.591110+0000","last_became_active":"2024-03-28T18:35:49.932207+0000","last_became_peered":"2024-03-28T18:35:49.932207+0000","last_unstale":"2024-03-28T18:36:54.591110+0000","last_undegraded":"2024-03-28T18:36:54.591110+0000","last_fullsized":"2024-03-28T18:36:54.591110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:50.606075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.522581+0000","last_change":"2024-03-28T18:35:50.696535+0000","last_active":"2024-03-28T18:37:23.522581+0000","last_peered":"2024-03-28T18:37:23.522581+0000","last_clean":"2024-03-28T18:37:23.522581+0000","last_became_active":"2024-03-28T18:35:50.696142+0000","last_became_peered":"2024-03-28T18:35:50.696142+0000","last_unstale":"2024-03-28T18:37:23.522581+0000","last_undegraded":"2024-03-28T18:37:23.522581+0000","last_fullsized":"2024-03-28T18:37:23.522581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:24:59.749302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596689+0000","last_change":"2024-03-28T18:36:54.596689+0000","last_active":"2024-03-28T18:36:54.596689+0000","last_peered":"2024-03-28T18:36:54.596689+0000","last_clean":"2024-03-28T18:36:54.596689+0000","last_became_active":"2024-03-28T18:36:54.595693+0000","last_became_peered":"2024-03-28T18:36:54.595693+0000","last_unstale":"2024-03-28T18:36:54.596689+0000","last_undegraded":"2024-03-28T18:36:54.596689+0000","last_fullsized":"2024-03-28T18:36:54.596689+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:20.698835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590946+0000","last_change":"2024-03-28T18:35:49.929798+0000","last_active":"2024-03-28T18:36:54.590946+0000","last_peered":"2024-03-28T18:36:54.590946+0000","last_clean":"2024-03-28T18:36:54.590946+0000","last_became_active":"2024-03-28T18:35:49.929278+0000","last_became_peered":"2024-03-28T18:35:49.929278+0000","last_unstale":"2024-03-28T18:36:54.590946+0000","last_undegraded":"2024-03-28T18:36:54.590946+0000","last_fullsized":"2024-03-28T18:36:54.590946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:33:13.078225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521904+0000","last_change":"2024-03-28T18:35:50.702935+0000","last_active":"2024-03-28T18:37:23.521904+0000","last_peered":"2024-03-28T18:37:23.521904+0000","last_clean":"2024-03-28T18:37:23.521904+0000","last_became_active":"2024-03-28T18:35:50.702586+0000","last_became_peered":"2024-03-28T18:35:50.702586+0000","last_unstale":"2024-03-28T18:37:23.521904+0000","last_undegraded":"2024-03-28T18:37:23.521904+0000","last_fullsized":"2024-03-28T18:37:23.521904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:58:50.335605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594102+0000","last_change":"2024-03-28T18:35:55.028541+0000","last_active":"2024-03-28T18:36:54.594102+0000","last_peered":"2024-03-28T18:36:54.594102+0000","last_clean":"2024-03-28T18:36:54.594102+0000","last_became_active":"2024-03-28T18:35:55.028142+0000","last_became_peered":"2024-03-28T18:35:55.028142+0000","last_unstale":"2024-03-28T18:36:54.594102+0000","last_undegraded":"2024-03-28T18:36:54.594102+0000","last_fullsized":"2024-03-28T18:36:54.594102+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:20.738071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588676+0000","last_change":"2024-03-28T18:35:49.938255+0000","last_active":"2024-03-28T18:36:54.588676+0000","last_peered":"2024-03-28T18:36:54.588676+0000","last_clean":"2024-03-28T18:36:54.588676+0000","last_became_active":"2024-03-28T18:35:49.937930+0000","last_became_peered":"2024-03-28T18:35:49.937930+0000","last_unstale":"2024-03-28T18:36:54.588676+0000","last_undegraded":"2024-03-28T18:36:54.588676+0000","last_fullsized":"2024-03-28T18:36:54.588676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:41.621941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.504921+0000","last_change":"2024-03-28T18:35:50.706437+0000","last_active":"2024-03-28T18:37:23.504921+0000","last_peered":"2024-03-28T18:37:23.504921+0000","last_clean":"2024-03-28T18:37:23.504921+0000","last_became_active":"2024-03-28T18:35:50.706139+0000","last_became_peered":"2024-03-28T18:35:50.706139+0000","last_unstale":"2024-03-28T18:37:23.504921+0000","last_undegraded":"2024-03-28T18:37:23.504921+0000","last_fullsized":"2024-03-28T18:37:23.504921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:46:52.750896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.598151+0000","last_change":"2024-03-28T18:36:54.598151+0000","last_active":"2024-03-28T18:36:54.598151+0000","last_peered":"2024-03-28T18:36:54.598151+0000","last_clean":"2024-03-28T18:36:54.598151+0000","last_became_active":"2024-03-28T18:36:54.597664+0000","last_became_peered":"2024-03-28T18:36:54.597664+0000","last_unstale":"2024-03-28T18:36:54.598151+0000","last_undegraded":"2024-03-28T18:36:54.598151+0000","last_fullsized":"2024-03-28T18:36:54.598151+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:23:34.772747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,8],"acting":[3,7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":179,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:25.089438+0000","last_change":"2024-03-28T18:35:49.940518+0000","last_active":"2024-03-28T18:37:25.089438+0000","last_peered":"2024-03-28T18:37:25.089438+0000","last_clean":"2024-03-28T18:37:25.089438+0000","last_became_active":"2024-03-28T18:35:49.940208+0000","last_became_peered":"2024-03-28T18:35:49.940208+0000","last_unstale":"2024-03-28T18:37:25.089438+0000","last_undegraded":"2024-03-28T18:37:25.089438+0000","last_fullsized":"2024-03-28T18:37:25.089438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:55.557840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515829+0000","last_change":"2024-03-28T18:35:50.710781+0000","last_active":"2024-03-28T18:37:23.515829+0000","last_peered":"2024-03-28T18:37:23.515829+0000","last_clean":"2024-03-28T18:37:23.515829+0000","last_became_active":"2024-03-28T18:35:50.710458+0000","last_became_peered":"2024-03-28T18:35:50.710458+0000","last_unstale":"2024-03-28T18:37:23.515829+0000","last_undegraded":"2024-03-28T18:37:23.515829+0000","last_fullsized":"2024-03-28T18:37:23.515829+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:35:49.020331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601148+0000","last_change":"2024-03-28T18:35:55.034023+0000","last_active":"2024-03-28T18:36:54.601148+0000","last_peered":"2024-03-28T18:36:54.601148+0000","last_clean":"2024-03-28T18:36:54.601148+0000","last_became_active":"2024-03-28T18:35:55.033634+0000","last_became_peered":"2024-03-28T18:35:55.033634+0000","last_unstale":"2024-03-28T18:36:54.601148+0000","last_undegraded":"2024-03-28T18:36:54.601148+0000","last_fullsized":"2024-03-28T18:36:54.601148+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:41.221061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.603273+0000","last_change":"2024-03-28T18:36:54.603273+0000","last_active":"2024-03-28T18:36:54.603273+0000","last_peered":"2024-03-28T18:36:54.603273+0000","last_clean":"2024-03-28T18:36:54.603273+0000","last_became_active":"2024-03-28T18:36:54.596666+0000","last_became_peered":"2024-03-28T18:36:54.596666+0000","last_unstale":"2024-03-28T18:36:54.603273+0000","last_undegraded":"2024-03-28T18:36:54.603273+0000","last_fullsized":"2024-03-28T18:36:54.603273+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:17.480746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.532308+0000","last_change":"2024-03-28T18:35:50.718480+0000","last_active":"2024-03-28T18:37:23.532308+0000","last_peered":"2024-03-28T18:37:23.532308+0000","last_clean":"2024-03-28T18:37:23.532308+0000","last_became_active":"2024-03-28T18:35:50.717253+0000","last_became_peered":"2024-03-28T18:35:50.717253+0000","last_unstale":"2024-03-28T18:37:23.532308+0000","last_undegraded":"2024-03-28T18:37:23.532308+0000","last_fullsized":"2024-03-28T18:37:23.532308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:59:30.071806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589384+0000","last_change":"2024-03-28T18:35:55.032461+0000","last_active":"2024-03-28T18:36:54.589384+0000","last_peered":"2024-03-28T18:36:54.589384+0000","last_clean":"2024-03-28T18:36:54.589384+0000","last_became_active":"2024-03-28T18:35:55.032029+0000","last_became_peered":"2024-03-28T18:35:55.032029+0000","last_unstale":"2024-03-28T18:36:54.589384+0000","last_undegraded":"2024-03-28T18:36:54.589384+0000","last_fullsized":"2024-03-28T18:36:54.589384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:27:11.216371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589446+0000","last_change":"2024-03-28T18:35:49.934357+0000","last_active":"2024-03-28T18:36:54.589446+0000","last_peered":"2024-03-28T18:36:54.589446+0000","last_clean":"2024-03-28T18:36:54.589446+0000","last_became_active":"2024-03-28T18:35:49.933994+0000","last_became_peered":"2024-03-28T18:35:49.933994+0000","last_unstale":"2024-03-28T18:36:54.589446+0000","last_undegraded":"2024-03-28T18:36:54.589446+0000","last_fullsized":"2024-03-28T18:36:54.589446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:21.408585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514262+0000","last_change":"2024-03-28T18:35:50.716771+0000","last_active":"2024-03-28T18:37:23.514262+0000","last_peered":"2024-03-28T18:37:23.514262+0000","last_clean":"2024-03-28T18:37:23.514262+0000","last_became_active":"2024-03-28T18:35:50.716393+0000","last_became_peered":"2024-03-28T18:35:50.716393+0000","last_unstale":"2024-03-28T18:37:23.514262+0000","last_undegraded":"2024-03-28T18:37:23.514262+0000","last_fullsized":"2024-03-28T18:37:23.514262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:33:10.881475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.594854+0000","last_change":"2024-03-28T18:36:54.594281+0000","last_active":"2024-03-28T18:36:54.594854+0000","last_peered":"2024-03-28T18:36:54.594854+0000","last_clean":"2024-03-28T18:36:54.594854+0000","last_became_active":"2024-03-28T18:36:54.593846+0000","last_became_peered":"2024-03-28T18:36:54.593846+0000","last_unstale":"2024-03-28T18:36:54.594854+0000","last_undegraded":"2024-03-28T18:36:54.594854+0000","last_fullsized":"2024-03-28T18:36:54.594854+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:51.911167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'35","reported_seq":218,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:25.092066+0000","last_change":"2024-03-28T18:35:49.934499+0000","last_active":"2024-03-28T18:37:25.092066+0000","last_peered":"2024-03-28T18:37:25.092066+0000","last_clean":"2024-03-28T18:37:25.092066+0000","last_became_active":"2024-03-28T18:35:49.934150+0000","last_became_peered":"2024-03-28T18:35:49.934150+0000","last_unstale":"2024-03-28T18:37:25.092066+0000","last_undegraded":"2024-03-28T18:37:25.092066+0000","last_fullsized":"2024-03-28T18:37:25.092066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:53.661593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61400,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":156,"num_read_kb":138,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.491098+0000","last_change":"2024-03-28T18:35:50.714162+0000","last_active":"2024-03-28T18:37:23.491098+0000","last_peered":"2024-03-28T18:37:23.491098+0000","last_clean":"2024-03-28T18:37:23.491098+0000","last_became_active":"2024-03-28T18:35:50.713819+0000","last_became_peered":"2024-03-28T18:35:50.713819+0000","last_unstale":"2024-03-28T18:37:23.491098+0000","last_undegraded":"2024-03-28T18:37:23.491098+0000","last_fullsized":"2024-03-28T18:37:23.491098+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:49:19.365559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589676+0000","last_change":"2024-03-28T18:35:55.028460+0000","last_active":"2024-03-28T18:36:54.589676+0000","last_peered":"2024-03-28T18:36:54.589676+0000","last_clean":"2024-03-28T18:36:54.589676+0000","last_became_active":"2024-03-28T18:35:55.028015+0000","last_became_peered":"2024-03-28T18:35:55.028015+0000","last_unstale":"2024-03-28T18:36:54.589676+0000","last_undegraded":"2024-03-28T18:36:54.589676+0000","last_fullsized":"2024-03-28T18:36:54.589676+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:33:00.864611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.596300+0000","last_change":"2024-03-28T18:35:49.938090+0000","last_active":"2024-03-28T18:36:54.596300+0000","last_peered":"2024-03-28T18:36:54.596300+0000","last_clean":"2024-03-28T18:36:54.596300+0000","last_became_active":"2024-03-28T18:35:49.937572+0000","last_became_peered":"2024-03-28T18:35:49.937572+0000","last_unstale":"2024-03-28T18:36:54.596300+0000","last_undegraded":"2024-03-28T18:36:54.596300+0000","last_fullsized":"2024-03-28T18:36:54.596300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:41.782246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.517300+0000","last_change":"2024-03-28T18:35:50.717286+0000","last_active":"2024-03-28T18:37:23.517300+0000","last_peered":"2024-03-28T18:37:23.517300+0000","last_clean":"2024-03-28T18:37:23.517300+0000","last_became_active":"2024-03-28T18:35:50.712917+0000","last_became_peered":"2024-03-28T18:35:50.712917+0000","last_unstale":"2024-03-28T18:37:23.517300+0000","last_undegraded":"2024-03-28T18:37:23.517300+0000","last_fullsized":"2024-03-28T18:37:23.517300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:10:42.306028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590424+0000","last_change":"2024-03-28T18:35:55.023384+0000","last_active":"2024-03-28T18:36:54.590424+0000","last_peered":"2024-03-28T18:36:54.590424+0000","last_clean":"2024-03-28T18:36:54.590424+0000","last_became_active":"2024-03-28T18:35:55.023013+0000","last_became_peered":"2024-03-28T18:35:55.023013+0000","last_unstale":"2024-03-28T18:36:54.590424+0000","last_undegraded":"2024-03-28T18:36:54.590424+0000","last_fullsized":"2024-03-28T18:36:54.590424+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:54:01.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589045+0000","last_change":"2024-03-28T18:35:49.939073+0000","last_active":"2024-03-28T18:36:54.589045+0000","last_peered":"2024-03-28T18:36:54.589045+0000","last_clean":"2024-03-28T18:36:54.589045+0000","last_became_active":"2024-03-28T18:35:49.938764+0000","last_became_peered":"2024-03-28T18:35:49.938764+0000","last_unstale":"2024-03-28T18:36:54.589045+0000","last_undegraded":"2024-03-28T18:36:54.589045+0000","last_fullsized":"2024-03-28T18:36:54.589045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:52.089059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.516846+0000","last_change":"2024-03-28T18:35:50.716132+0000","last_active":"2024-03-28T18:37:23.516846+0000","last_peered":"2024-03-28T18:37:23.516846+0000","last_clean":"2024-03-28T18:37:23.516846+0000","last_became_active":"2024-03-28T18:35:50.715768+0000","last_became_peered":"2024-03-28T18:35:50.715768+0000","last_unstale":"2024-03-28T18:37:23.516846+0000","last_undegraded":"2024-03-28T18:37:23.516846+0000","last_fullsized":"2024-03-28T18:37:23.516846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:42:57.930739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601576+0000","last_change":"2024-03-28T18:35:55.035553+0000","last_active":"2024-03-28T18:36:54.601576+0000","last_peered":"2024-03-28T18:36:54.601576+0000","last_clean":"2024-03-28T18:36:54.601576+0000","last_became_active":"2024-03-28T18:35:55.035134+0000","last_became_peered":"2024-03-28T18:35:55.035134+0000","last_unstale":"2024-03-28T18:36:54.601576+0000","last_undegraded":"2024-03-28T18:36:54.601576+0000","last_fullsized":"2024-03-28T18:36:54.601576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:03:52.976361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:27.102724+0000","last_change":"2024-03-28T18:35:49.932696+0000","last_active":"2024-03-28T18:37:27.102724+0000","last_peered":"2024-03-28T18:37:27.102724+0000","last_clean":"2024-03-28T18:37:27.102724+0000","last_became_active":"2024-03-28T18:35:49.932350+0000","last_became_peered":"2024-03-28T18:35:49.932350+0000","last_unstale":"2024-03-28T18:37:27.102724+0000","last_undegraded":"2024-03-28T18:37:27.102724+0000","last_fullsized":"2024-03-28T18:37:27.102724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:26.183549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.500849+0000","last_change":"2024-03-28T18:35:50.712734+0000","last_active":"2024-03-28T18:37:23.500849+0000","last_peered":"2024-03-28T18:37:23.500849+0000","last_clean":"2024-03-28T18:37:23.500849+0000","last_became_active":"2024-03-28T18:35:50.712409+0000","last_became_peered":"2024-03-28T18:35:50.712409+0000","last_unstale":"2024-03-28T18:37:23.500849+0000","last_undegraded":"2024-03-28T18:37:23.500849+0000","last_fullsized":"2024-03-28T18:37:23.500849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:06:36.344182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591837+0000","last_change":"2024-03-28T18:35:55.025047+0000","last_active":"2024-03-28T18:36:54.591837+0000","last_peered":"2024-03-28T18:36:54.591837+0000","last_clean":"2024-03-28T18:36:54.591837+0000","last_became_active":"2024-03-28T18:35:55.024652+0000","last_became_peered":"2024-03-28T18:35:55.024652+0000","last_unstale":"2024-03-28T18:36:54.591837+0000","last_undegraded":"2024-03-28T18:36:54.591837+0000","last_fullsized":"2024-03-28T18:36:54.591837+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:58:17.750951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.614368+0000","last_change":"2024-03-28T18:36:54.613995+0000","last_active":"2024-03-28T18:36:54.614368+0000","last_peered":"2024-03-28T18:36:54.614368+0000","last_clean":"2024-03-28T18:36:54.614368+0000","last_became_active":"2024-03-28T18:36:54.599838+0000","last_became_peered":"2024-03-28T18:36:54.599838+0000","last_unstale":"2024-03-28T18:36:54.614368+0000","last_undegraded":"2024-03-28T18:36:54.614368+0000","last_fullsized":"2024-03-28T18:36:54.614368+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:02.855413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.508762+0000","last_change":"2024-03-28T18:35:50.705023+0000","last_active":"2024-03-28T18:37:23.508762+0000","last_peered":"2024-03-28T18:37:23.508762+0000","last_clean":"2024-03-28T18:37:23.508762+0000","last_became_active":"2024-03-28T18:35:50.704694+0000","last_became_peered":"2024-03-28T18:35:50.704694+0000","last_unstale":"2024-03-28T18:37:23.508762+0000","last_undegraded":"2024-03-28T18:37:23.508762+0000","last_fullsized":"2024-03-28T18:37:23.508762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:45:46.677189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588854+0000","last_change":"2024-03-28T18:35:55.028311+0000","last_active":"2024-03-28T18:36:54.588854+0000","last_peered":"2024-03-28T18:36:54.588854+0000","last_clean":"2024-03-28T18:36:54.588854+0000","last_became_active":"2024-03-28T18:35:55.027914+0000","last_became_peered":"2024-03-28T18:35:55.027914+0000","last_unstale":"2024-03-28T18:36:54.588854+0000","last_undegraded":"2024-03-28T18:36:54.588854+0000","last_fullsized":"2024-03-28T18:36:54.588854+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:59.684230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593503+0000","last_change":"2024-03-28T18:35:49.938004+0000","last_active":"2024-03-28T18:36:54.593503+0000","last_peered":"2024-03-28T18:36:54.593503+0000","last_clean":"2024-03-28T18:36:54.593503+0000","last_became_active":"2024-03-28T18:35:49.937615+0000","last_became_peered":"2024-03-28T18:35:49.937615+0000","last_unstale":"2024-03-28T18:36:54.593503+0000","last_undegraded":"2024-03-28T18:36:54.593503+0000","last_fullsized":"2024-03-28T18:36:54.593503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:21.619857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503409+0000","last_change":"2024-03-28T18:35:50.719276+0000","last_active":"2024-03-28T18:37:23.503409+0000","last_peered":"2024-03-28T18:37:23.503409+0000","last_clean":"2024-03-28T18:37:23.503409+0000","last_became_active":"2024-03-28T18:35:50.718896+0000","last_became_peered":"2024-03-28T18:35:50.718896+0000","last_unstale":"2024-03-28T18:37:23.503409+0000","last_undegraded":"2024-03-28T18:37:23.503409+0000","last_fullsized":"2024-03-28T18:37:23.503409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:24.494713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592412+0000","last_change":"2024-03-28T18:35:55.099193+0000","last_active":"2024-03-28T18:36:54.592412+0000","last_peered":"2024-03-28T18:36:54.592412+0000","last_clean":"2024-03-28T18:36:54.592412+0000","last_became_active":"2024-03-28T18:35:55.098795+0000","last_became_peered":"2024-03-28T18:35:55.098795+0000","last_unstale":"2024-03-28T18:36:54.592412+0000","last_undegraded":"2024-03-28T18:36:54.592412+0000","last_fullsized":"2024-03-28T18:36:54.592412+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:07.872554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601410+0000","last_change":"2024-03-28T18:35:49.936885+0000","last_active":"2024-03-28T18:36:54.601410+0000","last_peered":"2024-03-28T18:36:54.601410+0000","last_clean":"2024-03-28T18:36:54.601410+0000","last_became_active":"2024-03-28T18:35:49.936563+0000","last_became_peered":"2024-03-28T18:35:49.936563+0000","last_unstale":"2024-03-28T18:36:54.601410+0000","last_undegraded":"2024-03-28T18:36:54.601410+0000","last_fullsized":"2024-03-28T18:36:54.601410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:22:33.631160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514866+0000","last_change":"2024-03-28T18:35:50.713750+0000","last_active":"2024-03-28T18:37:23.514866+0000","last_peered":"2024-03-28T18:37:23.514866+0000","last_clean":"2024-03-28T18:37:23.514866+0000","last_became_active":"2024-03-28T18:35:50.713378+0000","last_became_peered":"2024-03-28T18:35:50.713378+0000","last_unstale":"2024-03-28T18:37:23.514866+0000","last_undegraded":"2024-03-28T18:37:23.514866+0000","last_fullsized":"2024-03-28T18:37:23.514866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:36.072651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591101+0000","last_change":"2024-03-28T18:35:55.030827+0000","last_active":"2024-03-28T18:36:54.591101+0000","last_peered":"2024-03-28T18:36:54.591101+0000","last_clean":"2024-03-28T18:36:54.591101+0000","last_became_active":"2024-03-28T18:35:55.030382+0000","last_became_peered":"2024-03-28T18:35:55.030382+0000","last_unstale":"2024-03-28T18:36:54.591101+0000","last_undegraded":"2024-03-28T18:36:54.591101+0000","last_fullsized":"2024-03-28T18:36:54.591101+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:41:58.371568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":109,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:27.117944+0000","last_change":"2024-03-28T18:35:49.932290+0000","last_active":"2024-03-28T18:37:27.117944+0000","last_peered":"2024-03-28T18:37:27.117944+0000","last_clean":"2024-03-28T18:37:27.117944+0000","last_became_active":"2024-03-28T18:35:49.931925+0000","last_became_peered":"2024-03-28T18:35:49.931925+0000","last_unstale":"2024-03-28T18:37:27.117944+0000","last_undegraded":"2024-03-28T18:37:27.117944+0000","last_fullsized":"2024-03-28T18:37:27.117944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:12:26.477117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.492659+0000","last_change":"2024-03-28T18:35:50.713654+0000","last_active":"2024-03-28T18:37:23.492659+0000","last_peered":"2024-03-28T18:37:23.492659+0000","last_clean":"2024-03-28T18:37:23.492659+0000","last_became_active":"2024-03-28T18:35:50.713146+0000","last_became_peered":"2024-03-28T18:35:50.713146+0000","last_unstale":"2024-03-28T18:37:23.492659+0000","last_undegraded":"2024-03-28T18:37:23.492659+0000","last_fullsized":"2024-03-28T18:37:23.492659+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:23.951963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591861+0000","last_change":"2024-03-28T18:35:55.099331+0000","last_active":"2024-03-28T18:36:54.591861+0000","last_peered":"2024-03-28T18:36:54.591861+0000","last_clean":"2024-03-28T18:36:54.591861+0000","last_became_active":"2024-03-28T18:35:55.098912+0000","last_became_peered":"2024-03-28T18:35:55.098912+0000","last_unstale":"2024-03-28T18:36:54.591861+0000","last_undegraded":"2024-03-28T18:36:54.591861+0000","last_fullsized":"2024-03-28T18:36:54.591861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:31.082148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506783+0000","last_change":"2024-03-28T18:35:50.704377+0000","last_active":"2024-03-28T18:37:23.506783+0000","last_peered":"2024-03-28T18:37:23.506783+0000","last_clean":"2024-03-28T18:37:23.506783+0000","last_became_active":"2024-03-28T18:35:50.703991+0000","last_became_peered":"2024-03-28T18:35:50.703991+0000","last_unstale":"2024-03-28T18:37:23.506783+0000","last_undegraded":"2024-03-28T18:37:23.506783+0000","last_fullsized":"2024-03-28T18:37:23.506783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:48.173466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:26.101292+0000","last_change":"2024-03-28T18:35:49.933528+0000","last_active":"2024-03-28T18:37:26.101292+0000","last_peered":"2024-03-28T18:37:26.101292+0000","last_clean":"2024-03-28T18:37:26.101292+0000","last_became_active":"2024-03-28T18:35:49.933171+0000","last_became_peered":"2024-03-28T18:35:49.933171+0000","last_unstale":"2024-03-28T18:37:26.101292+0000","last_undegraded":"2024-03-28T18:37:26.101292+0000","last_fullsized":"2024-03-28T18:37:26.101292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:50:59.928400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593337+0000","last_change":"2024-03-28T18:35:55.029752+0000","last_active":"2024-03-28T18:36:54.593337+0000","last_peered":"2024-03-28T18:36:54.593337+0000","last_clean":"2024-03-28T18:36:54.593337+0000","last_became_active":"2024-03-28T18:35:55.029407+0000","last_became_peered":"2024-03-28T18:35:55.029407+0000","last_unstale":"2024-03-28T18:36:54.593337+0000","last_undegraded":"2024-03-28T18:36:54.593337+0000","last_fullsized":"2024-03-28T18:36:54.593337+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:17.328315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.519894+0000","last_change":"2024-03-28T18:35:50.706380+0000","last_active":"2024-03-28T18:37:23.519894+0000","last_peered":"2024-03-28T18:37:23.519894+0000","last_clean":"2024-03-28T18:37:23.519894+0000","last_became_active":"2024-03-28T18:35:50.706044+0000","last_became_peered":"2024-03-28T18:35:50.706044+0000","last_unstale":"2024-03-28T18:37:23.519894+0000","last_undegraded":"2024-03-28T18:37:23.519894+0000","last_fullsized":"2024-03-28T18:37:23.519894+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:04.916646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589278+0000","last_change":"2024-03-28T18:35:49.930241+0000","last_active":"2024-03-28T18:36:54.589278+0000","last_peered":"2024-03-28T18:36:54.589278+0000","last_clean":"2024-03-28T18:36:54.589278+0000","last_became_active":"2024-03-28T18:35:49.929830+0000","last_became_peered":"2024-03-28T18:35:49.929830+0000","last_unstale":"2024-03-28T18:36:54.589278+0000","last_undegraded":"2024-03-28T18:36:54.589278+0000","last_fullsized":"2024-03-28T18:36:54.589278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:11.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592511+0000","last_change":"2024-03-28T18:34:38.804796+0000","last_active":"2024-03-28T18:36:54.592511+0000","last_peered":"2024-03-28T18:36:54.592511+0000","last_clean":"2024-03-28T18:36:54.592511+0000","last_became_active":"2024-03-28T18:34:38.804419+0000","last_became_peered":"2024-03-28T18:34:38.804419+0000","last_unstale":"2024-03-28T18:36:54.592511+0000","last_undegraded":"2024-03-28T18:36:54.592511+0000","last_fullsized":"2024-03-28T18:36:54.592511+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:31:13.733610+0000","last_clean_scrub_stamp":"2024-03-28T18:31:13.733610+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:33:38.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589683+0000","last_change":"2024-03-28T18:35:55.099390+0000","last_active":"2024-03-28T18:36:54.589683+0000","last_peered":"2024-03-28T18:36:54.589683+0000","last_clean":"2024-03-28T18:36:54.589683+0000","last_became_active":"2024-03-28T18:35:55.098960+0000","last_became_peered":"2024-03-28T18:35:55.098960+0000","last_unstale":"2024-03-28T18:36:54.589683+0000","last_undegraded":"2024-03-28T18:36:54.589683+0000","last_fullsized":"2024-03-28T18:36:54.589683+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:22.460846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503603+0000","last_change":"2024-03-28T18:35:50.711961+0000","last_active":"2024-03-28T18:37:23.503603+0000","last_peered":"2024-03-28T18:37:23.503603+0000","last_clean":"2024-03-28T18:37:23.503603+0000","last_became_active":"2024-03-28T18:35:50.705466+0000","last_became_peered":"2024-03-28T18:35:50.705466+0000","last_unstale":"2024-03-28T18:37:23.503603+0000","last_undegraded":"2024-03-28T18:37:23.503603+0000","last_fullsized":"2024-03-28T18:37:23.503603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:20:02.527220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.603568+0000","last_change":"2024-03-28T18:36:54.602798+0000","last_active":"2024-03-28T18:36:54.603568+0000","last_peered":"2024-03-28T18:36:54.603568+0000","last_clean":"2024-03-28T18:36:54.603568+0000","last_became_active":"2024-03-28T18:36:54.599176+0000","last_became_peered":"2024-03-28T18:36:54.599176+0000","last_unstale":"2024-03-28T18:36:54.603568+0000","last_undegraded":"2024-03-28T18:36:54.603568+0000","last_fullsized":"2024-03-28T18:36:54.603568+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:27:17.065687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590414+0000","last_change":"2024-03-28T18:35:55.023402+0000","last_active":"2024-03-28T18:36:54.590414+0000","last_peered":"2024-03-28T18:36:54.590414+0000","last_clean":"2024-03-28T18:36:54.590414+0000","last_became_active":"2024-03-28T18:35:55.023006+0000","last_became_peered":"2024-03-28T18:35:55.023006+0000","last_unstale":"2024-03-28T18:36:54.590414+0000","last_undegraded":"2024-03-28T18:36:54.590414+0000","last_fullsized":"2024-03-28T18:36:54.590414+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:12.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592600+0000","last_change":"2024-03-28T18:35:49.942494+0000","last_active":"2024-03-28T18:36:54.592600+0000","last_peered":"2024-03-28T18:36:54.592600+0000","last_clean":"2024-03-28T18:36:54.592600+0000","last_became_active":"2024-03-28T18:35:49.941607+0000","last_became_peered":"2024-03-28T18:35:49.941607+0000","last_unstale":"2024-03-28T18:36:54.592600+0000","last_undegraded":"2024-03-28T18:36:54.592600+0000","last_fullsized":"2024-03-28T18:36:54.592600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:25.315100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.504331+0000","last_change":"2024-03-28T18:35:50.707912+0000","last_active":"2024-03-28T18:37:23.504331+0000","last_peered":"2024-03-28T18:37:23.504331+0000","last_clean":"2024-03-28T18:37:23.504331+0000","last_became_active":"2024-03-28T18:35:50.707304+0000","last_became_peered":"2024-03-28T18:35:50.707304+0000","last_unstale":"2024-03-28T18:37:23.504331+0000","last_undegraded":"2024-03-28T18:37:23.504331+0000","last_fullsized":"2024-03-28T18:37:23.504331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:45.556370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590548+0000","last_change":"2024-03-28T18:35:55.099919+0000","last_active":"2024-03-28T18:36:54.590548+0000","last_peered":"2024-03-28T18:36:54.590548+0000","last_clean":"2024-03-28T18:36:54.590548+0000","last_became_active":"2024-03-28T18:35:55.099430+0000","last_became_peered":"2024-03-28T18:35:55.099430+0000","last_unstale":"2024-03-28T18:36:54.590548+0000","last_undegraded":"2024-03-28T18:36:54.590548+0000","last_fullsized":"2024-03-28T18:36:54.590548+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:13:30.091473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601614+0000","last_change":"2024-03-28T18:35:49.935871+0000","last_active":"2024-03-28T18:36:54.601614+0000","last_peered":"2024-03-28T18:36:54.601614+0000","last_clean":"2024-03-28T18:36:54.601614+0000","last_became_active":"2024-03-28T18:35:49.935536+0000","last_became_peered":"2024-03-28T18:35:49.935536+0000","last_unstale":"2024-03-28T18:36:54.601614+0000","last_undegraded":"2024-03-28T18:36:54.601614+0000","last_fullsized":"2024-03-28T18:36:54.601614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:26:22.915067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.524489+0000","last_change":"2024-03-28T18:35:50.710877+0000","last_active":"2024-03-28T18:37:23.524489+0000","last_peered":"2024-03-28T18:37:23.524489+0000","last_clean":"2024-03-28T18:37:23.524489+0000","last_became_active":"2024-03-28T18:35:50.709987+0000","last_became_peered":"2024-03-28T18:35:50.709987+0000","last_unstale":"2024-03-28T18:37:23.524489+0000","last_undegraded":"2024-03-28T18:37:23.524489+0000","last_fullsized":"2024-03-28T18:37:23.524489+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:42:43.092541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595346+0000","last_change":"2024-03-28T18:35:55.023390+0000","last_active":"2024-03-28T18:36:54.595346+0000","last_peered":"2024-03-28T18:36:54.595346+0000","last_clean":"2024-03-28T18:36:54.595346+0000","last_became_active":"2024-03-28T18:35:55.023034+0000","last_became_peered":"2024-03-28T18:35:55.023034+0000","last_unstale":"2024-03-28T18:36:54.595346+0000","last_undegraded":"2024-03-28T18:36:54.595346+0000","last_fullsized":"2024-03-28T18:36:54.595346+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:32:05.829144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.590255+0000","last_change":"2024-03-28T18:35:49.932675+0000","last_active":"2024-03-28T18:36:54.590255+0000","last_peered":"2024-03-28T18:36:54.590255+0000","last_clean":"2024-03-28T18:36:54.590255+0000","last_became_active":"2024-03-28T18:35:49.932246+0000","last_became_peered":"2024-03-28T18:35:49.932246+0000","last_unstale":"2024-03-28T18:36:54.590255+0000","last_undegraded":"2024-03-28T18:36:54.590255+0000","last_fullsized":"2024-03-28T18:36:54.590255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:37:46.806411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.531219+0000","last_change":"2024-03-28T18:35:50.716861+0000","last_active":"2024-03-28T18:37:23.531219+0000","last_peered":"2024-03-28T18:37:23.531219+0000","last_clean":"2024-03-28T18:37:23.531219+0000","last_became_active":"2024-03-28T18:35:50.716377+0000","last_became_peered":"2024-03-28T18:35:50.716377+0000","last_unstale":"2024-03-28T18:37:23.531219+0000","last_undegraded":"2024-03-28T18:37:23.531219+0000","last_fullsized":"2024-03-28T18:37:23.531219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:57:27.018809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589916+0000","last_change":"2024-03-28T18:35:55.099891+0000","last_active":"2024-03-28T18:36:54.589916+0000","last_peered":"2024-03-28T18:36:54.589916+0000","last_clean":"2024-03-28T18:36:54.589916+0000","last_became_active":"2024-03-28T18:35:55.098988+0000","last_became_peered":"2024-03-28T18:35:55.098988+0000","last_unstale":"2024-03-28T18:36:54.589916+0000","last_undegraded":"2024-03-28T18:36:54.589916+0000","last_fullsized":"2024-03-28T18:36:54.589916+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:32:15.682731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592671+0000","last_change":"2024-03-28T18:35:49.936146+0000","last_active":"2024-03-28T18:36:54.592671+0000","last_peered":"2024-03-28T18:36:54.592671+0000","last_clean":"2024-03-28T18:36:54.592671+0000","last_became_active":"2024-03-28T18:35:49.935771+0000","last_became_peered":"2024-03-28T18:35:49.935771+0000","last_unstale":"2024-03-28T18:36:54.592671+0000","last_undegraded":"2024-03-28T18:36:54.592671+0000","last_fullsized":"2024-03-28T18:36:54.592671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:58.840229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.485426+0000","last_change":"2024-03-28T18:35:50.707527+0000","last_active":"2024-03-28T18:37:23.485426+0000","last_peered":"2024-03-28T18:37:23.485426+0000","last_clean":"2024-03-28T18:37:23.485426+0000","last_became_active":"2024-03-28T18:35:50.707216+0000","last_became_peered":"2024-03-28T18:35:50.707216+0000","last_unstale":"2024-03-28T18:37:23.485426+0000","last_undegraded":"2024-03-28T18:37:23.485426+0000","last_fullsized":"2024-03-28T18:37:23.485426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:32.344898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.601648+0000","last_change":"2024-03-28T18:35:55.033294+0000","last_active":"2024-03-28T18:36:54.601648+0000","last_peered":"2024-03-28T18:36:54.601648+0000","last_clean":"2024-03-28T18:36:54.601648+0000","last_became_active":"2024-03-28T18:35:55.030743+0000","last_became_peered":"2024-03-28T18:35:55.030743+0000","last_unstale":"2024-03-28T18:36:54.601648+0000","last_undegraded":"2024-03-28T18:36:54.601648+0000","last_fullsized":"2024-03-28T18:36:54.601648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:16:11.003542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592174+0000","last_change":"2024-03-28T18:35:49.931394+0000","last_active":"2024-03-28T18:36:54.592174+0000","last_peered":"2024-03-28T18:36:54.592174+0000","last_clean":"2024-03-28T18:36:54.592174+0000","last_became_active":"2024-03-28T18:35:49.931045+0000","last_became_peered":"2024-03-28T18:35:49.931045+0000","last_unstale":"2024-03-28T18:36:54.592174+0000","last_undegraded":"2024-03-28T18:36:54.592174+0000","last_fullsized":"2024-03-28T18:36:54.592174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:54.030481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.522605+0000","last_change":"2024-03-28T18:35:50.714745+0000","last_active":"2024-03-28T18:37:23.522605+0000","last_peered":"2024-03-28T18:37:23.522605+0000","last_clean":"2024-03-28T18:37:23.522605+0000","last_became_active":"2024-03-28T18:35:50.712991+0000","last_became_peered":"2024-03-28T18:35:50.712991+0000","last_unstale":"2024-03-28T18:37:23.522605+0000","last_undegraded":"2024-03-28T18:37:23.522605+0000","last_fullsized":"2024-03-28T18:37:23.522605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:34.403662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589455+0000","last_change":"2024-03-28T18:35:55.032353+0000","last_active":"2024-03-28T18:36:54.589455+0000","last_peered":"2024-03-28T18:36:54.589455+0000","last_clean":"2024-03-28T18:36:54.589455+0000","last_became_active":"2024-03-28T18:35:55.031889+0000","last_became_peered":"2024-03-28T18:35:55.031889+0000","last_unstale":"2024-03-28T18:36:54.589455+0000","last_undegraded":"2024-03-28T18:36:54.589455+0000","last_fullsized":"2024-03-28T18:36:54.589455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:56.358941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.595619+0000","last_change":"2024-03-28T18:35:49.944107+0000","last_active":"2024-03-28T18:36:54.595619+0000","last_peered":"2024-03-28T18:36:54.595619+0000","last_clean":"2024-03-28T18:36:54.595619+0000","last_became_active":"2024-03-28T18:35:49.943726+0000","last_became_peered":"2024-03-28T18:35:49.943726+0000","last_unstale":"2024-03-28T18:36:54.595619+0000","last_undegraded":"2024-03-28T18:36:54.595619+0000","last_fullsized":"2024-03-28T18:36:54.595619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:12:05.301302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.532032+0000","last_change":"2024-03-28T18:35:50.709236+0000","last_active":"2024-03-28T18:37:23.532032+0000","last_peered":"2024-03-28T18:37:23.532032+0000","last_clean":"2024-03-28T18:37:23.532032+0000","last_became_active":"2024-03-28T18:35:50.708824+0000","last_became_peered":"2024-03-28T18:35:50.708824+0000","last_unstale":"2024-03-28T18:37:23.532032+0000","last_undegraded":"2024-03-28T18:37:23.532032+0000","last_fullsized":"2024-03-28T18:37:23.532032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:21:30.207033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589738+0000","last_change":"2024-03-28T18:35:55.099711+0000","last_active":"2024-03-28T18:36:54.589738+0000","last_peered":"2024-03-28T18:36:54.589738+0000","last_clean":"2024-03-28T18:36:54.589738+0000","last_became_active":"2024-03-28T18:35:55.099054+0000","last_became_peered":"2024-03-28T18:35:55.099054+0000","last_unstale":"2024-03-28T18:36:54.589738+0000","last_undegraded":"2024-03-28T18:36:54.589738+0000","last_fullsized":"2024-03-28T18:36:54.589738+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:22:32.326655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592694+0000","last_change":"2024-03-28T18:35:49.939428+0000","last_active":"2024-03-28T18:36:54.592694+0000","last_peered":"2024-03-28T18:36:54.592694+0000","last_clean":"2024-03-28T18:36:54.592694+0000","last_became_active":"2024-03-28T18:35:49.939089+0000","last_became_peered":"2024-03-28T18:35:49.939089+0000","last_unstale":"2024-03-28T18:36:54.592694+0000","last_undegraded":"2024-03-28T18:36:54.592694+0000","last_fullsized":"2024-03-28T18:36:54.592694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:51:38.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.515324+0000","last_change":"2024-03-28T18:35:50.720931+0000","last_active":"2024-03-28T18:37:23.515324+0000","last_peered":"2024-03-28T18:37:23.515324+0000","last_clean":"2024-03-28T18:37:23.515324+0000","last_became_active":"2024-03-28T18:35:50.720544+0000","last_became_peered":"2024-03-28T18:35:50.720544+0000","last_unstale":"2024-03-28T18:37:23.515324+0000","last_undegraded":"2024-03-28T18:37:23.515324+0000","last_fullsized":"2024-03-28T18:37:23.515324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:05.092688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.592432+0000","last_change":"2024-03-28T18:35:55.024518+0000","last_active":"2024-03-28T18:36:54.592432+0000","last_peered":"2024-03-28T18:36:54.592432+0000","last_clean":"2024-03-28T18:36:54.592432+0000","last_became_active":"2024-03-28T18:35:55.024103+0000","last_became_peered":"2024-03-28T18:35:55.024103+0000","last_unstale":"2024-03-28T18:36:54.592432+0000","last_undegraded":"2024-03-28T18:36:54.592432+0000","last_fullsized":"2024-03-28T18:36:54.592432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:32:57.620744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'5","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.625690+0000","last_change":"2024-03-28T18:36:54.602349+0000","last_active":"2024-03-28T18:37:23.625690+0000","last_peered":"2024-03-28T18:37:23.625690+0000","last_clean":"2024-03-28T18:37:23.625690+0000","last_became_active":"2024-03-28T18:36:54.595886+0000","last_became_peered":"2024-03-28T18:36:54.595886+0000","last_unstale":"2024-03-28T18:37:23.625690+0000","last_undegraded":"2024-03-28T18:37:23.625690+0000","last_fullsized":"2024-03-28T18:37:23.625690+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:25:02.437159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.508021+0000","last_change":"2024-03-28T18:35:50.720449+0000","last_active":"2024-03-28T18:37:23.508021+0000","last_peered":"2024-03-28T18:37:23.508021+0000","last_clean":"2024-03-28T18:37:23.508021+0000","last_became_active":"2024-03-28T18:35:50.719840+0000","last_became_peered":"2024-03-28T18:35:50.719840+0000","last_unstale":"2024-03-28T18:37:23.508021+0000","last_undegraded":"2024-03-28T18:37:23.508021+0000","last_fullsized":"2024-03-28T18:37:23.508021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:48.980677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.593710+0000","last_change":"2024-03-28T18:35:55.032614+0000","last_active":"2024-03-28T18:36:54.593710+0000","last_peered":"2024-03-28T18:36:54.593710+0000","last_clean":"2024-03-28T18:36:54.593710+0000","last_became_active":"2024-03-28T18:35:55.032180+0000","last_became_peered":"2024-03-28T18:35:55.032180+0000","last_unstale":"2024-03-28T18:36:54.593710+0000","last_undegraded":"2024-03-28T18:36:54.593710+0000","last_fullsized":"2024-03-28T18:36:54.593710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:07:40.121016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.589786+0000","last_change":"2024-03-28T18:35:49.940681+0000","last_active":"2024-03-28T18:36:54.589786+0000","last_peered":"2024-03-28T18:36:54.589786+0000","last_clean":"2024-03-28T18:36:54.589786+0000","last_became_active":"2024-03-28T18:35:49.940376+0000","last_became_peered":"2024-03-28T18:35:49.940376+0000","last_unstale":"2024-03-28T18:36:54.589786+0000","last_undegraded":"2024-03-28T18:36:54.589786+0000","last_fullsized":"2024-03-28T18:36:54.589786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:02:11.926285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506994+0000","last_change":"2024-03-28T18:35:50.715051+0000","last_active":"2024-03-28T18:37:23.506994+0000","last_peered":"2024-03-28T18:37:23.506994+0000","last_clean":"2024-03-28T18:37:23.506994+0000","last_became_active":"2024-03-28T18:35:50.714647+0000","last_became_peered":"2024-03-28T18:35:50.714647+0000","last_unstale":"2024-03-28T18:37:23.506994+0000","last_undegraded":"2024-03-28T18:37:23.506994+0000","last_fullsized":"2024-03-28T18:37:23.506994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:02.832665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.588600+0000","last_change":"2024-03-28T18:35:55.021869+0000","last_active":"2024-03-28T18:36:54.588600+0000","last_peered":"2024-03-28T18:36:54.588600+0000","last_clean":"2024-03-28T18:36:54.588600+0000","last_became_active":"2024-03-28T18:35:55.021535+0000","last_became_peered":"2024-03-28T18:35:55.021535+0000","last_unstale":"2024-03-28T18:36:54.588600+0000","last_undegraded":"2024-03-28T18:36:54.588600+0000","last_fullsized":"2024-03-28T18:36:54.588600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:53.999615+0000","last_clean_scrub_stamp":"2024-03-28T18:35:53.999615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:59:31.610882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:36:54.591586+0000","last_change":"2024-03-28T18:35:49.936095+0000","last_active":"2024-03-28T18:36:54.591586+0000","last_peered":"2024-03-28T18:36:54.591586+0000","last_clean":"2024-03-28T18:36:54.591586+0000","last_became_active":"2024-03-28T18:35:49.935766+0000","last_became_peered":"2024-03-28T18:35:49.935766+0000","last_unstale":"2024-03-28T18:36:54.591586+0000","last_undegraded":"2024-03-28T18:36:54.591586+0000","last_fullsized":"2024-03-28T18:36:54.591586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:48.907270+0000","last_clean_scrub_stamp":"2024-03-28T18:35:48.907270+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:00:06.712979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.507860+0000","last_change":"2024-03-28T18:35:50.715313+0000","last_active":"2024-03-28T18:37:23.507860+0000","last_peered":"2024-03-28T18:37:23.507860+0000","last_clean":"2024-03-28T18:37:23.507860+0000","last_became_active":"2024-03-28T18:35:50.714923+0000","last_became_peered":"2024-03-28T18:35:50.714923+0000","last_unstale":"2024-03-28T18:37:23.507860+0000","last_undegraded":"2024-03-28T18:37:23.507860+0000","last_fullsized":"2024-03-28T18:37:23.507860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:42:49.406551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.504412+0000","last_change":"2024-03-28T18:35:50.714244+0000","last_active":"2024-03-28T18:37:23.504412+0000","last_peered":"2024-03-28T18:37:23.504412+0000","last_clean":"2024-03-28T18:37:23.504412+0000","last_became_active":"2024-03-28T18:35:50.713347+0000","last_became_peered":"2024-03-28T18:35:50.713347+0000","last_unstale":"2024-03-28T18:37:23.504412+0000","last_undegraded":"2024-03-28T18:37:23.504412+0000","last_fullsized":"2024-03-28T18:37:23.504412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:19:34.360341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.505208+0000","last_change":"2024-03-28T18:35:50.711132+0000","last_active":"2024-03-28T18:37:23.505208+0000","last_peered":"2024-03-28T18:37:23.505208+0000","last_clean":"2024-03-28T18:37:23.505208+0000","last_became_active":"2024-03-28T18:35:50.710500+0000","last_became_peered":"2024-03-28T18:35:50.710500+0000","last_unstale":"2024-03-28T18:37:23.505208+0000","last_undegraded":"2024-03-28T18:37:23.505208+0000","last_fullsized":"2024-03-28T18:37:23.505208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:51:22.060106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.529114+0000","last_change":"2024-03-28T18:35:50.712396+0000","last_active":"2024-03-28T18:37:23.529114+0000","last_peered":"2024-03-28T18:37:23.529114+0000","last_clean":"2024-03-28T18:37:23.529114+0000","last_became_active":"2024-03-28T18:35:50.711950+0000","last_became_peered":"2024-03-28T18:35:50.711950+0000","last_unstale":"2024-03-28T18:37:23.529114+0000","last_undegraded":"2024-03-28T18:37:23.529114+0000","last_fullsized":"2024-03-28T18:37:23.529114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:47:24.573754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.529027+0000","last_change":"2024-03-28T18:35:50.718222+0000","last_active":"2024-03-28T18:37:23.529027+0000","last_peered":"2024-03-28T18:37:23.529027+0000","last_clean":"2024-03-28T18:37:23.529027+0000","last_became_active":"2024-03-28T18:35:50.717234+0000","last_became_peered":"2024-03-28T18:35:50.717234+0000","last_unstale":"2024-03-28T18:37:23.529027+0000","last_undegraded":"2024-03-28T18:37:23.529027+0000","last_fullsized":"2024-03-28T18:37:23.529027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:28:47.544280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.528568+0000","last_change":"2024-03-28T18:35:50.707486+0000","last_active":"2024-03-28T18:37:23.528568+0000","last_peered":"2024-03-28T18:37:23.528568+0000","last_clean":"2024-03-28T18:37:23.528568+0000","last_became_active":"2024-03-28T18:35:50.706902+0000","last_became_peered":"2024-03-28T18:35:50.706902+0000","last_unstale":"2024-03-28T18:37:23.528568+0000","last_undegraded":"2024-03-28T18:37:23.528568+0000","last_fullsized":"2024-03-28T18:37:23.528568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:19:00.160873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.503369+0000","last_change":"2024-03-28T18:35:50.715566+0000","last_active":"2024-03-28T18:37:23.503369+0000","last_peered":"2024-03-28T18:37:23.503369+0000","last_clean":"2024-03-28T18:37:23.503369+0000","last_became_active":"2024-03-28T18:35:50.715193+0000","last_became_peered":"2024-03-28T18:35:50.715193+0000","last_unstale":"2024-03-28T18:37:23.503369+0000","last_undegraded":"2024-03-28T18:37:23.503369+0000","last_fullsized":"2024-03-28T18:37:23.503369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:57.126250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.514038+0000","last_change":"2024-03-28T18:35:50.709904+0000","last_active":"2024-03-28T18:37:23.514038+0000","last_peered":"2024-03-28T18:37:23.514038+0000","last_clean":"2024-03-28T18:37:23.514038+0000","last_became_active":"2024-03-28T18:35:50.709575+0000","last_became_peered":"2024-03-28T18:35:50.709575+0000","last_unstale":"2024-03-28T18:37:23.514038+0000","last_undegraded":"2024-03-28T18:37:23.514038+0000","last_fullsized":"2024-03-28T18:37:23.514038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:18:26.580718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.519901+0000","last_change":"2024-03-28T18:36:54.598585+0000","last_active":"2024-03-28T18:37:23.519901+0000","last_peered":"2024-03-28T18:37:23.519901+0000","last_clean":"2024-03-28T18:37:23.519901+0000","last_became_active":"2024-03-28T18:36:54.598262+0000","last_became_peered":"2024-03-28T18:36:54.598262+0000","last_unstale":"2024-03-28T18:37:23.519901+0000","last_undegraded":"2024-03-28T18:37:23.519901+0000","last_fullsized":"2024-03-28T18:37:23.519901+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:16:46.028970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521405+0000","last_change":"2024-03-28T18:35:50.707563+0000","last_active":"2024-03-28T18:37:23.521405+0000","last_peered":"2024-03-28T18:37:23.521405+0000","last_clean":"2024-03-28T18:37:23.521405+0000","last_became_active":"2024-03-28T18:35:50.706300+0000","last_became_peered":"2024-03-28T18:35:50.706300+0000","last_unstale":"2024-03-28T18:37:23.521405+0000","last_undegraded":"2024-03-28T18:37:23.521405+0000","last_fullsized":"2024-03-28T18:37:23.521405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:05.324365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.524287+0000","last_change":"2024-03-28T18:35:50.705549+0000","last_active":"2024-03-28T18:37:23.524287+0000","last_peered":"2024-03-28T18:37:23.524287+0000","last_clean":"2024-03-28T18:37:23.524287+0000","last_became_active":"2024-03-28T18:35:50.705190+0000","last_became_peered":"2024-03-28T18:35:50.705190+0000","last_unstale":"2024-03-28T18:37:23.524287+0000","last_undegraded":"2024-03-28T18:37:23.524287+0000","last_fullsized":"2024-03-28T18:37:23.524287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:15:38.662227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.521023+0000","last_change":"2024-03-28T18:35:50.719591+0000","last_active":"2024-03-28T18:37:23.521023+0000","last_peered":"2024-03-28T18:37:23.521023+0000","last_clean":"2024-03-28T18:37:23.521023+0000","last_became_active":"2024-03-28T18:35:50.719272+0000","last_became_peered":"2024-03-28T18:35:50.719272+0000","last_unstale":"2024-03-28T18:37:23.521023+0000","last_undegraded":"2024-03-28T18:37:23.521023+0000","last_fullsized":"2024-03-28T18:37:23.521023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:06.627761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.532242+0000","last_change":"2024-03-28T18:35:50.712965+0000","last_active":"2024-03-28T18:37:23.532242+0000","last_peered":"2024-03-28T18:37:23.532242+0000","last_clean":"2024-03-28T18:37:23.532242+0000","last_became_active":"2024-03-28T18:35:50.712648+0000","last_became_peered":"2024-03-28T18:35:50.712648+0000","last_unstale":"2024-03-28T18:37:23.532242+0000","last_undegraded":"2024-03-28T18:37:23.532242+0000","last_fullsized":"2024-03-28T18:37:23.532242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:00.925664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-28T18:37:23.506395+0000","last_change":"2024-03-28T18:35:50.717277+0000","last_active":"2024-03-28T18:37:23.506395+0000","last_peered":"2024-03-28T18:37:23.506395+0000","last_clean":"2024-03-28T18:37:23.506395+0000","last_became_active":"2024-03-28T18:35:50.716161+0000","last_became_peered":"2024-03-28T18:35:50.716161+0000","last_unstale":"2024-03-28T18:37:23.506395+0000","last_undegraded":"2024-03-28T18:37:23.506395+0000","last_fullsized":"2024-03-28T18:37:23.506395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:35:49.909358+0000","last_clean_scrub_stamp":"2024-03-28T18:35:49.909358+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:27:42.789575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":210208,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68354,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":741,"num_read_kb":462,"num_write":247,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1282,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":205062,"data_compressed":4791,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318304,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":511389,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1171456,"data_stored":1160025,"data_compressed":57783,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743467,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":515303,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005104,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":511473,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1101824,"data_stored":1108223,"data_compressed":56186,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":506711,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":570311,"data_compressed":22201,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":508167,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":568191,"data_compressed":22201,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":506419,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":511317,"data_compressed":20604,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":1103273,"data_compressed":56186,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33940,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35582,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64322,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62202,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63668,"data_compressed":1597,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:37:30.991 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T18:37:30.991 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T18:37:30.992 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T18:37:30.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T18:37:31.246 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:31.682 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:31 smithi145 ceph-mon[32151]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:31.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:31 smithi192 ceph-mon[36589]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:31 smithi164 ceph-mon[36248]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:37:32.688 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:32 smithi145 ceph-mon[32151]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:32.807 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:32 smithi192 ceph-mon[36589]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:32 smithi164 ceph-mon[36248]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:33.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 -- 172.21.15.145:0/386178233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c801081e0 msgr2=0x7f4c801085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:33.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 --2- 172.21.15.145:0/386178233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c801081e0 0x7f4c801085c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4c740098f0 tx=0x7f4c74009c00 comp rx=0 tx=0).stop 2024-03-28T18:37:33.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 -- 172.21.15.145:0/386178233 shutdown_connections 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 --2- 172.21.15.145:0/386178233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c8006a7e0 0x7f4c8010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 --2- 172.21.15.145:0/386178233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c801081e0 0x7f4c801085c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 --2- 172.21.15.145:0/386178233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c80069e40 0x7f4c8006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 -- 172.21.15.145:0/386178233 >> 172.21.15.145:0/386178233 conn(0x7f4c80077fc0 msgr2=0x7f4c800783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 -- 172.21.15.145:0/386178233 shutdown_connections 2024-03-28T18:37:33.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.050+0000 7f4c86af7700 1 -- 172.21.15.145:0/386178233 wait complete. 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 Processor -- start 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 -- start start 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 0x7f4c8019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 0x7f4c801987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:33.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c80112440 con 0x7f4c80069e40 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c801122c0 con 0x7f4c8006a7e0 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c86af7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c801125c0 con 0x7f4c801081e0 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c84893700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c84893700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59538/0 (socket says 172.21.15.145:59538) 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c84893700 1 -- 172.21.15.145:0/1416185951 learned_addr learned my addr 172.21.15.145:0/1416185951 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c7ffff700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 0x7f4c8019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.051+0000 7f4c85094700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 0x7f4c801987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 -- 172.21.15.145:0/1416185951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 msgr2=0x7f4c801987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 0x7f4c801987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 -- 172.21.15.145:0/1416185951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 msgr2=0x7f4c8019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 0x7f4c8019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 -- 172.21.15.145:0/1416185951 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c801990a0 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c85094700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 0x7f4c801987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c7ffff700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 0x7f4c8019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c84893700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009230 tx=0x7f4c7400ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74005570 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c80199330 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c7400ba40 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74017400 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.052+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c801a5450 con 0x7f4c80069e40 2024-03-28T18:37:33.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.053+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c80103890 con 0x7f4c80069e40 2024-03-28T18:37:33.057 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.055+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4c74027060 con 0x7f4c80069e40 2024-03-28T18:37:33.057 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.056+0000 7f4c7dffb700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 0x7f4c700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:33.057 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.056+0000 7f4c7ffff700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 0x7f4c700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:33.057 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.056+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f4c740cf610 con 0x7f4c80069e40 2024-03-28T18:37:33.058 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.057+0000 7f4c7ffff700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 0x7f4c700a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4c80199f30 tx=0x7f4c6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:33.059 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.059+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c7405b740 con 0x7f4c80069e40 2024-03-28T18:37:33.311 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.310+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c8002da30 con 0x7f4c80069e40 2024-03-28T18:37:33.311 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.310+0000 7f4c7dffb700 1 -- 172.21.15.145:0/1416185951 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4c74013030 con 0x7f4c80069e40 2024-03-28T18:37:33.311 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:33.312 INFO:teuthology.orchestra.run.smithi145.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T18:37:33.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 msgr2=0x7f4c700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:33.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 0x7f4c700a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4c80199f30 tx=0x7f4c6c009260 comp rx=0 tx=0).stop 2024-03-28T18:37:33.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 msgr2=0x7f4c8019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009230 tx=0x7f4c7400ff20 comp rx=0 tx=0).stop 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 shutdown_connections 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4c700a2890 0x7f4c700a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4c801081e0 0x7f4c801987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c80069e40 0x7f4c8019e6a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 --2- 172.21.15.145:0/1416185951 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c8006a7e0 0x7f4c8019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.313+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 >> 172.21.15.145:0/1416185951 conn(0x7f4c80077fc0 msgr2=0x7f4c800ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.314+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 shutdown_connections 2024-03-28T18:37:33.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:33.314+0000 7f4c86af7700 1 -- 172.21.15.145:0/1416185951 wait complete. 2024-03-28T18:37:33.621 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:33 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1416185951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:37:33.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:33 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1416185951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:37:33.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:33 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1416185951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:37:34.083 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T18:37:34.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:37:34.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.189+0000 7fb63d030640 1 -- 172.21.15.145:0/2587124003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 msgr2=0x7fb638108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.189+0000 7fb63d030640 1 --2- 172.21.15.145:0/2587124003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb638108a00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb630009ca0 tx=0x7fb63002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 -- 172.21.15.145:0/2587124003 shutdown_connections 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 --2- 172.21.15.145:0/2587124003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb638108a00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 --2- 172.21.15.145:0/2587124003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381017b0 0x7fb638101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 --2- 172.21.15.145:0/2587124003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381005c0 0x7fb6381009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 -- 172.21.15.145:0/2587124003 >> 172.21.15.145:0/2587124003 conn(0x7fb6380fbb50 msgr2=0x7fb6380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 -- 172.21.15.145:0/2587124003 shutdown_connections 2024-03-28T18:37:34.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 -- 172.21.15.145:0/2587124003 wait complete. 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 Processor -- start 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.190+0000 7fb63d030640 1 -- start start 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 0x7fb6381979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 0x7fb638197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63810c240 con 0x7fb638102170 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63810c0c0 con 0x7fb6381017b0 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63d030640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63810c3c0 con 0x7fb6381005c0 2024-03-28T18:37:34.193 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63759e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb636d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 0x7fb6381979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 0x7fb638197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63759e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59558/0 (socket says 172.21.15.145:59558) 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63759e640 1 -- 172.21.15.145:0/1097877987 learned_addr learned my addr 172.21.15.145:0/1097877987 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb636d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 0x7fb6381979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52108/0 (socket says 172.21.15.145:52108) 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 -- 172.21.15.145:0/1097877987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 msgr2=0x7fb6381979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 0x7fb6381979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 -- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 msgr2=0x7fb63819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 -- 172.21.15.145:0/1097877987 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63819f5d0 con 0x7fb6381017b0 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63759e640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:34.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb63659c640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 0x7fb638197ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00e9f0 tx=0x7fb62c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.191+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00ccc0 con 0x7fb6381017b0 2024-03-28T18:37:34.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.192+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63819f8c0 con 0x7fb6381017b0 2024-03-28T18:37:34.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.192+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb62c005020 con 0x7fb6381017b0 2024-03-28T18:37:34.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.192+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0049e0 con 0x7fb6381017b0 2024-03-28T18:37:34.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.192+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63819fdc0 con 0x7fb6381017b0 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.193+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6381a0080 con 0x7fb6381017b0 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.194+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb62c004c40 con 0x7fb6381017b0 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.194+0000 7fb61ffff640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 0x7fb6200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.194+0000 7fb636d9d640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 0x7fb6200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.194+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb62c0ced60 con 0x7fb6381017b0 2024-03-28T18:37:34.196 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.195+0000 7fb636d9d640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 0x7fb6200a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb638101610 tx=0x7fb628005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.200 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.199+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb62c09baa0 con 0x7fb6381017b0 2024-03-28T18:37:34.383 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.382+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6381a0720 con 0x7fb6381017b0 2024-03-28T18:37:34.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.385+0000 7fb61ffff640 1 -- 172.21.15.145:0/1097877987 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24703 (secure 0 0 0) 0x7fb62c0051d0 con 0x7fb6381017b0 2024-03-28T18:37:34.386 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:34.386 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":93,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:36:54.585370+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.387+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 msgr2=0x7fb6200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.387+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 0x7fb6200a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb638101610 tx=0x7fb628005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 msgr2=0x7fb638197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 0x7fb638197ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00e9f0 tx=0x7fb62c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 shutdown_connections 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb638102170 0x7fb63819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6381017b0 0x7fb638197ef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb6200a2800 0x7fb6200a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 --2- 172.21.15.145:0/1097877987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6381005c0 0x7fb6381979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 >> 172.21.15.145:0/1097877987 conn(0x7fb6380fbb50 msgr2=0x7fb6380fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 shutdown_connections 2024-03-28T18:37:34.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.388+0000 7fb63d030640 1 -- 172.21.15.145:0/1097877987 wait complete. 2024-03-28T18:37:34.400 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:37:34.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- 172.21.15.145:0/3494148188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 msgr2=0x7f5f301089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- 172.21.15.145:0/3494148188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301089e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009ca0 tx=0x7f5f2802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- 172.21.15.145:0/3494148188 shutdown_connections 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- 172.21.15.145:0/3494148188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301089e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- 172.21.15.145:0/3494148188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f30101790 0x7f5f30101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- 172.21.15.145:0/3494148188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f301005a0 0x7f5f301009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- 172.21.15.145:0/3494148188 >> 172.21.15.145:0/3494148188 conn(0x7f5f300fbb30 msgr2=0x7f5f300fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:34.513 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- 172.21.15.145:0/3494148188 shutdown_connections 2024-03-28T18:37:34.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- 172.21.15.145:0/3494148188 wait complete. 2024-03-28T18:37:34.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 Processor -- start 2024-03-28T18:37:34.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 -- start start 2024-03-28T18:37:34.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f301005a0 0x7f5f30199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f30101790 0x7f5f3019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.512+0000 7f5f36322640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f36322640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3010c230 con 0x7f5f30102150 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f36322640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3010c0b0 con 0x7f5f30101790 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f36322640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3010c3b0 con 0x7f5f301005a0 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59578/0 (socket says 172.21.15.145:59578) 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f2ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f301005a0 0x7f5f30199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 -- 172.21.15.145:0/2794405333 learned_addr learned my addr 172.21.15.145:0/2794405333 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f2f7fe640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f30101790 0x7f5f3019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 -- 172.21.15.145:0/2794405333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f301005a0 msgr2=0x7f5f30199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f301005a0 0x7f5f30199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 -- 172.21.15.145:0/2794405333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f30101790 msgr2=0x7f5f3019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f30101790 0x7f5f3019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 -- 172.21.15.145:0/2794405333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f301a1780 con 0x7f5f30102150 2024-03-28T18:37:34.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f348bf640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f28037040 tx=0x7f5f28032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28041020 con 0x7f5f30102150 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f28034cd0 con 0x7f5f30102150 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f301a1a10 con 0x7f5f30102150 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28039040 con 0x7f5f30102150 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.513+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f301a1f00 con 0x7f5f30102150 2024-03-28T18:37:34.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.514+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f301a2130 con 0x7f5f30102150 2024-03-28T18:37:34.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.516+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5f28044020 con 0x7f5f30102150 2024-03-28T18:37:34.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.516+0000 7f5f2d7fa640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 0x7f5f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.517+0000 7f5f2ffff640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 0x7f5f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.517+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f5f280f10a0 con 0x7f5f30102150 2024-03-28T18:37:34.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.517+0000 7f5f2ffff640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 0x7f5f180a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f301015f0 tx=0x7f5f20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.521 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.520+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f2804b020 con 0x7f5f30102150 2024-03-28T18:37:34.682 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:34 smithi145 ceph-mon[32151]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:37:34.682 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:34 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1097877987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:34.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.707+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f301a24c0 con 0x7f5f30102150 2024-03-28T18:37:34.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.707+0000 7f5f2d7fa640 1 -- 172.21.15.145:0/2794405333 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f280bde70 con 0x7f5f30102150 2024-03-28T18:37:34.708 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:37:34.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 msgr2=0x7f5f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 0x7f5f180a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f301015f0 tx=0x7f5f20005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 msgr2=0x7f5f301a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f28037040 tx=0x7f5f28032f40 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 shutdown_connections 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f30102150 0x7f5f301a1110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f30101790 0x7f5f3019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5f180a2800 0x7f5f180a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 --2- 172.21.15.145:0/2794405333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f301005a0 0x7f5f30199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 >> 172.21.15.145:0/2794405333 conn(0x7f5f300fbb30 msgr2=0x7f5f300fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 shutdown_connections 2024-03-28T18:37:34.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.710+0000 7f5f36322640 1 -- 172.21.15.145:0/2794405333 wait complete. 2024-03-28T18:37:34.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:37:34.832 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 -- 172.21.15.145:0/4082872149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 msgr2=0x7ff30c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.832 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 --2- 172.21.15.145:0/4082872149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c1009c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff308009900 tx=0x7ff30802bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:34.832 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 -- 172.21.15.145:0/4082872149 shutdown_connections 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 --2- 172.21.15.145:0/4082872149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 0x7ff30c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 --2- 172.21.15.145:0/4082872149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 0x7ff30c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 --2- 172.21.15.145:0/4082872149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c1009c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 -- 172.21.15.145:0/4082872149 >> 172.21.15.145:0/4082872149 conn(0x7ff30c0fbb30 msgr2=0x7ff30c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 -- 172.21.15.145:0/4082872149 shutdown_connections 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.829+0000 7ff3141d3640 1 -- 172.21.15.145:0/4082872149 wait complete. 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 Processor -- start 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 -- start start 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 0x7ff30c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 0x7ff30c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c10c230 con 0x7ff30c1005a0 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c10c0b0 con 0x7ff30c102150 2024-03-28T18:37:34.833 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff3141d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c10c3b0 con 0x7ff30c101790 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59604/0 (socket says 172.21.15.145:59604) 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 -- 172.21.15.145:0/3396095353 learned_addr learned my addr 172.21.15.145:0/3396095353 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff31176e640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 0x7ff30c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff312770640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 0x7ff30c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 -- 172.21.15.145:0/3396095353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 msgr2=0x7ff30c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 0x7ff30c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 -- 172.21.15.145:0/3396095353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 msgr2=0x7ff30c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 0x7ff30c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.830+0000 7ff311f6f640 1 -- 172.21.15.145:0/3396095353 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c19f650 con 0x7ff30c1005a0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff311f6f640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff308000c00 tx=0x7ff30802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308041070 con 0x7ff30c1005a0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff308004450 con 0x7ff30c1005a0 2024-03-28T18:37:34.834 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c19f8e0 con 0x7ff30c1005a0 2024-03-28T18:37:34.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308034440 con 0x7ff30c1005a0 2024-03-28T18:37:34.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.831+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c19fdd0 con 0x7ff30c1005a0 2024-03-28T18:37:34.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.832+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c19ffd0 con 0x7ff30c1005a0 2024-03-28T18:37:34.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.833+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff308044070 con 0x7ff30c1005a0 2024-03-28T18:37:34.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.834+0000 7ff302ffd640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 0x7ff2f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:34.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.834+0000 7ff31176e640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 0x7ff2f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:34.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.834+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff3080f0a90 con 0x7ff30c1005a0 2024-03-28T18:37:34.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.834+0000 7ff31176e640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 0x7ff2f80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff30c198f40 tx=0x7ff2fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:34.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:34.838+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3080bd210 con 0x7ff30c1005a0 2024-03-28T18:37:34.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:34 smithi192 ceph-mon[36589]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:37:34.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:34 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1097877987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:34.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:34 smithi164 ceph-mon[36248]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:37:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:34 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1097877987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:35.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.015+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff30c1a01a0 con 0x7ff30c1005a0 2024-03-28T18:37:35.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.016+0000 7ff302ffd640 1 -- 172.21.15.145:0/3396095353 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3080c12b0 con 0x7ff30c1005a0 2024-03-28T18:37:35.017 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.017+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 msgr2=0x7ff2f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 0x7ff2f80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff30c198f40 tx=0x7ff2fc005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 msgr2=0x7ff30c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff308000c00 tx=0x7ff30802c640 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 shutdown_connections 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30c102150 0x7ff30c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff2f80a2800 0x7ff2f80a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff30c101790 0x7ff30c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 --2- 172.21.15.145:0/3396095353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff30c1005a0 0x7ff30c197a30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 >> 172.21.15.145:0/3396095353 conn(0x7ff30c0fbb30 msgr2=0x7ff30c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 shutdown_connections 2024-03-28T18:37:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.018+0000 7ff3141d3640 1 -- 172.21.15.145:0/3396095353 wait complete. 2024-03-28T18:37:35.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1826258889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c401005a0 msgr2=0x7f5c401009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1826258889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c401005a0 0x7f5c401009c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009900 tx=0x7f5c3c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1826258889 shutdown_connections 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1826258889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c40102150 0x7f5c401089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1826258889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 0x7f5c40101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1826258889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c401005a0 0x7f5c401009c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1826258889 >> 172.21.15.145:0/1826258889 conn(0x7f5c400fbb30 msgr2=0x7f5c400fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1826258889 shutdown_connections 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.142+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1826258889 wait complete. 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 Processor -- start 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 -- start start 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 0x7f5c40196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 0x7f5c40196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 0x7f5c4019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4010c230 con 0x7f5c40102150 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4010c0b0 con 0x7f5c401005a0 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c47e0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4010c3b0 con 0x7f5c40101790 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c463a8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 0x7f5c4019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c453a6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 0x7f5c40196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 0x7f5c40196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c463a8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 0x7f5c4019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59632/0 (socket says 172.21.15.145:59632) 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c463a8640 1 -- 172.21.15.145:0/1553012456 learned_addr learned my addr 172.21.15.145:0/1553012456 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 -- 172.21.15.145:0/1553012456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 msgr2=0x7f5c40196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 0x7f5c40196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 -- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 msgr2=0x7f5c4019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 0x7f5c4019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.143+0000 7f5c45ba7640 1 -- 172.21.15.145:0/1553012456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4019e570 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c45ba7640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 0x7f5c40196950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02c170 tx=0x7f5c3c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c041070 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4019e800 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c3c004450 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c003e50 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.144+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4019ecc0 con 0x7f5c401005a0 2024-03-28T18:37:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.145+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c401079e0 con 0x7f5c401005a0 2024-03-28T18:37:35.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.146+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5c3c044070 con 0x7f5c401005a0 2024-03-28T18:37:35.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.146+0000 7f5c36ffd640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 0x7f5c2c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.146+0000 7f5c453a6640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 0x7f5c2c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.147+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f5c3c0f0a60 con 0x7f5c401005a0 2024-03-28T18:37:35.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.147+0000 7f5c453a6640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 0x7f5c2c0a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c40197e60 tx=0x7f5c30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.150+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c3c0bd150 con 0x7f5c401005a0 2024-03-28T18:37:35.371 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.370+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c400622a0 con 0x7f5c401005a0 2024-03-28T18:37:35.371 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.370+0000 7f5c36ffd640 1 -- 172.21.15.145:0/1553012456 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c3c0c11f0 con 0x7f5c401005a0 2024-03-28T18:37:35.371 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:37:35.373 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 msgr2=0x7f5c2c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 0x7f5c2c0a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c40197e60 tx=0x7f5c30005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 msgr2=0x7f5c40196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 0x7f5c40196950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02c170 tx=0x7f5c3c02c650 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 shutdown_connections 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c40102150 0x7f5c4019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5c2c0a29a0 0x7f5c2c0a4e60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c40101790 0x7f5c40196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 --2- 172.21.15.145:0/1553012456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c401005a0 0x7f5c40196950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.372+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 >> 172.21.15.145:0/1553012456 conn(0x7f5c400fbb30 msgr2=0x7f5c40104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.373+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 shutdown_connections 2024-03-28T18:37:35.374 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.373+0000 7f5c47e0b640 1 -- 172.21.15.145:0/1553012456 wait complete. 2024-03-28T18:37:35.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 -- 172.21.15.145:0/3554636115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1788101790 msgr2=0x7f1788101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 --2- 172.21.15.145:0/3554636115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1788101790 0x7f1788101c10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f177c009860 tx=0x7f177c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 -- 172.21.15.145:0/3554636115 shutdown_connections 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 --2- 172.21.15.145:0/3554636115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 --2- 172.21.15.145:0/3554636115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1788101790 0x7f1788101c10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 --2- 172.21.15.145:0/3554636115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17881005a0 0x7f17881009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 -- 172.21.15.145:0/3554636115 >> 172.21.15.145:0/3554636115 conn(0x7f17880fbb30 msgr2=0x7f17880fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 -- 172.21.15.145:0/3554636115 shutdown_connections 2024-03-28T18:37:35.491 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 -- 172.21.15.145:0/3554636115 wait complete. 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.490+0000 7f178d1f4640 1 Processor -- start 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 -- start start 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178810c230 con 0x7f1788101790 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178810c0b0 con 0x7f17881005a0 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178d1f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178810c3b0 con 0x7f1788102150 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51396/0 (socket says 172.21.15.145:51396) 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59648/0 (socket says 172.21.15.145:59648) 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f1786d9d640 1 -- 172.21.15.145:0/1966737706 learned_addr learned my addr 172.21.15.145:0/1966737706 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 -- 172.21.15.145:0/1966737706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 msgr2=0x7f1788069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 -- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 msgr2=0x7f178819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178759e640 1 -- 172.21.15.145:0/1966737706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17881a1800 con 0x7f1788102150 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f1786d9d640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.491+0000 7f178659c640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f178759e640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881a1030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000ce20 con 0x7f1788102150 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17881a1af0 con 0x7f1788102150 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17881a2060 con 0x7f1788102150 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1780010430 con 0x7f1788102150 2024-03-28T18:37:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.492+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178000f620 con 0x7f1788102150 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.493+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17881078c0 con 0x7f1788102150 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.494+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1780010950 con 0x7f1788102150 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.494+0000 7f176ffff640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 0x7f17700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.495+0000 7f1786d9d640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 0x7f17700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.495+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f17800cf4a0 con 0x7f1788102150 2024-03-28T18:37:35.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.495+0000 7f1786d9d640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 0x7f17700a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f17780096c0 tx=0x7f1778009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.502+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f178009c180 con 0x7f1788102150 2024-03-28T18:37:35.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.752+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f178802d070 con 0x7f1788102150 2024-03-28T18:37:35.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.752+0000 7f176ffff640 1 -- 172.21.15.145:0/1966737706 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f178009ba80 con 0x7f1788102150 2024-03-28T18:37:35.753 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:37:35.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 msgr2=0x7f17700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 0x7f17700a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f17780096c0 tx=0x7f1778009260 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 msgr2=0x7f17881a1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881a1030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1780009fd0 tx=0x7f178000ef10 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 shutdown_connections 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f17700a2800 0x7f17700a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1788102150 0x7f17881a1030 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1788101790 0x7f178819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 --2- 172.21.15.145:0/1966737706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17881005a0 0x7f1788069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 >> 172.21.15.145:0/1966737706 conn(0x7f17880fbb30 msgr2=0x7f1788104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 shutdown_connections 2024-03-28T18:37:35.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.754+0000 7f178d1f4640 1 -- 172.21.15.145:0/1966737706 wait complete. 2024-03-28T18:37:35.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:35 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2794405333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:35 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3396095353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.765 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:35 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1553012456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:37:35.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 -- 172.21.15.145:0/2760162333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe248102170 msgr2=0x7fe248108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/2760162333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe248102170 0x7fe248108a00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe240009ca0 tx=0x7fe24002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 -- 172.21.15.145:0/2760162333 shutdown_connections 2024-03-28T18:37:35.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/2760162333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe248102170 0x7fe248108a00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/2760162333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481017b0 0x7fe248101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/2760162333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2481005c0 0x7fe2481009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 -- 172.21.15.145:0/2760162333 >> 172.21.15.145:0/2760162333 conn(0x7fe2480fbb30 msgr2=0x7fe2480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 -- 172.21.15.145:0/2760162333 shutdown_connections 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.868+0000 7fe24f6c0640 1 -- 172.21.15.145:0/2760162333 wait complete. 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 Processor -- start 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 -- start start 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2481017b0 0x7fe24819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248102170 0x7fe24819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24810c240 con 0x7fe2481017b0 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24810c0c0 con 0x7fe248102170 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24f6c0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24810c3c0 con 0x7fe2481005c0 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24cc5b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2481017b0 0x7fe24819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52204/0 (socket says 172.21.15.145:52204) 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 -- 172.21.15.145:0/1391187891 learned_addr learned my addr 172.21.15.145:0/1391187891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24dc5d640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248102170 0x7fe24819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 -- 172.21.15.145:0/1391187891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248102170 msgr2=0x7fe24819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248102170 0x7fe24819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 -- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2481017b0 msgr2=0x7fe24819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2481017b0 0x7fe24819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.869+0000 7fe24d45c640 1 -- 172.21.15.145:0/1391187891 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24819f5e0 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe24d45c640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe24400e9c0 tx=0x7fe24400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24400cd80 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24819f870 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2440035f0 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244010690 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.870+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24819fe10 con 0x7fe2481005c0 2024-03-28T18:37:35.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.871+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe248107880 con 0x7fe2481005c0 2024-03-28T18:37:35.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.873+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2440037a0 con 0x7fe2481005c0 2024-03-28T18:37:35.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.873+0000 7fe23e7fc640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 0x7fe2340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:35.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.873+0000 7fe24cc5b640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 0x7fe2340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:35.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.873+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe2440cf250 con 0x7fe2481005c0 2024-03-28T18:37:35.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.874+0000 7fe24cc5b640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 0x7fe2340a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe2480690e0 tx=0x7fe238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:35.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:35.880+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe24409c020 con 0x7fe2481005c0 2024-03-28T18:37:35.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2794405333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3396095353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:35 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1553012456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:35 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2794405333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:35 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3396095353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:35.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:35 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1553012456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:36.123 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.123+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24802d070 con 0x7fe2481005c0 2024-03-28T18:37:36.124 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.124+0000 7fe23e7fc640 1 -- 172.21.15.145:0/1391187891 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7fe24409b920 con 0x7fe2481005c0 2024-03-28T18:37:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:36.125 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:37:36.125 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 msgr2=0x7fe2340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 0x7fe2340a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe2480690e0 tx=0x7fe238005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 msgr2=0x7fe248069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe24400e9c0 tx=0x7fe24400ee80 comp rx=0 tx=0).stop 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 shutdown_connections 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248102170 0x7fe24819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2481017b0 0x7fe24819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fe2340a2800 0x7fe2340a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 --2- 172.21.15.145:0/1391187891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe2481005c0 0x7fe248069370 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 >> 172.21.15.145:0/1391187891 conn(0x7fe2480fbb30 msgr2=0x7fe248104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 shutdown_connections 2024-03-28T18:37:36.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.126+0000 7fe24f6c0640 1 -- 172.21.15.145:0/1391187891 wait complete. 2024-03-28T18:37:36.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 -- 172.21.15.145:0/1663601464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 msgr2=0x7f0fec104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/1663601464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec104590 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009900 tx=0x7f0fe002bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 -- 172.21.15.145:0/1663601464 shutdown_connections 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/1663601464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 0x7f0fec1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/1663601464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 0x7f0fec106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/1663601464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec104590 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 -- 172.21.15.145:0/1663601464 >> 172.21.15.145:0/1663601464 conn(0x7f0fec0fbb50 msgr2=0x7f0fec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 -- 172.21.15.145:0/1663601464 shutdown_connections 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 -- 172.21.15.145:0/1663601464 wait complete. 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.240+0000 7f0ff2b09640 1 Processor -- start 2024-03-28T18:37:36.243 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 -- start start 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 0x7f0fec197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 0x7f0fec19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec10d360 con 0x7f0fec1021a0 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec10d1e0 con 0x7f0fec107400 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec10d4e0 con 0x7f0fec104ad0 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59690/0 (socket says 172.21.15.145:59690) 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 -- 172.21.15.145:0/2025147964 learned_addr learned my addr 172.21.15.145:0/2025147964 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0febfff640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 0x7f0fec197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff10a6640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 0x7f0fec19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 -- 172.21.15.145:0/2025147964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 msgr2=0x7f0fec197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 0x7f0fec197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 -- 172.21.15.145:0/2025147964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 msgr2=0x7f0fec19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.244 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 0x7f0fec19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 -- 172.21.15.145:0/2025147964 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec19f660 con 0x7f0fec1021a0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff08a5640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009230 tx=0x7f0fe002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0041070 con 0x7f0fec1021a0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec19f8f0 con 0x7f0fec1021a0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fe0004450 con 0x7f0fec1021a0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.241+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0003e20 con 0x7f0fec1021a0 2024-03-28T18:37:36.245 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.242+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec19fde0 con 0x7f0fec1021a0 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.243+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec0fd640 con 0x7f0fec1021a0 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.244+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0fe0044070 con 0x7f0fec1021a0 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.244+0000 7f0fe9ffb640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 0x7f0fdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.244+0000 7f0febfff640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 0x7f0fdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.244+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0fe00f0cf0 con 0x7f0fec1021a0 2024-03-28T18:37:36.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.245+0000 7f0febfff640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 0x7f0fdc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0fec198f50 tx=0x7f0fd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.249 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.249+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0fe00bd3e0 con 0x7f0fec1021a0 2024-03-28T18:37:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.431+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fec02d240 con 0x7f0fec1021a0 2024-03-28T18:37:36.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.433+0000 7f0fe9ffb640 1 -- 172.21.15.145:0/2025147964 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24703 (secure 0 0 0) 0x7f0fe00c1480 con 0x7f0fec1021a0 2024-03-28T18:37:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:36.434 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":93,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:36:54.585370+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:37:36.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 msgr2=0x7f0fdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 0x7f0fdc0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0fec198f50 tx=0x7f0fd8005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:36.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 msgr2=0x7f0fec197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009230 tx=0x7f0fe002c650 comp rx=0 tx=0).stop 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 shutdown_connections 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fec107400 0x7f0fec19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fdc0a2800 0x7f0fdc0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fec104ad0 0x7f0fec197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 --2- 172.21.15.145:0/2025147964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fec1021a0 0x7f0fec197a40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 >> 172.21.15.145:0/2025147964 conn(0x7f0fec0fbb50 msgr2=0x7f0fec102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 shutdown_connections 2024-03-28T18:37:36.437 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.435+0000 7f0ff2b09640 1 -- 172.21.15.145:0/2025147964 wait complete. 2024-03-28T18:37:36.448 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:37:36.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:37:36.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2802984964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 msgr2=0x7f3f441009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2802984964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f441009e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009860 tx=0x7f3f3402bb70 comp rx=0 tx=0).stop 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2802984964 shutdown_connections 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2802984964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f44102170 0x7f3f44108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2802984964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f441017b0 0x7f3f44101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2802984964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f441009e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2802984964 >> 172.21.15.145:0/2802984964 conn(0x7f3f440fbb30 msgr2=0x7f3f440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2802984964 shutdown_connections 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.553+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2802984964 wait complete. 2024-03-28T18:37:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 Processor -- start 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 -- start start 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f4406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 0x7f3f4419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4410c230 con 0x7f3f44102170 2024-03-28T18:37:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4410c0b0 con 0x7f3f441017b0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f48bdf640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4410c3b0 con 0x7f3f441005c0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.554+0000 7f3f42d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f42d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59704/0 (socket says 172.21.15.145:59704) 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f4406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f42d9d640 1 -- 172.21.15.145:0/2689893146 learned_addr learned my addr 172.21.15.145:0/2689893146 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f39d9b640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 0x7f3f4419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 -- 172.21.15.145:0/2689893146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 msgr2=0x7f3f4419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 0x7f3f4419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 -- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 msgr2=0x7f3f4419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 -- 172.21.15.145:0/2689893146 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4419f700 con 0x7f3f441005c0 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f42d9d640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f39d9b640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 0x7f3f4419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f4259c640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f4406a4b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402c0b0 tx=0x7f3f340323b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34041020 con 0x7f3f441005c0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f3402c9f0 con 0x7f3f441005c0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34039040 con 0x7f3f441005c0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4419f930 con 0x7f3f441005c0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.555+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4419fe20 con 0x7f3f441005c0 2024-03-28T18:37:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.556+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f44107a00 con 0x7f3f441005c0 2024-03-28T18:37:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.558+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3f34044020 con 0x7f3f441005c0 2024-03-28T18:37:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.558+0000 7f3f3bfff640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.558+0000 7f3f39d9b640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.558+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3f340f09e0 con 0x7f3f441005c0 2024-03-28T18:37:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.559+0000 7f3f39d9b640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 0x7f3f300a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0096c0 tx=0x7f3f2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.561+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f3404b020 con 0x7f3f441005c0 2024-03-28T18:37:36.746 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.745+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f4402d2c0 con 0x7f3f441005c0 2024-03-28T18:37:36.746 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.746+0000 7f3f3bfff640 1 -- 172.21.15.145:0/2689893146 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f340bd6c0 con 0x7f3f441005c0 2024-03-28T18:37:36.746 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 msgr2=0x7f3f300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 0x7f3f300a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0096c0 tx=0x7f3f2c009260 comp rx=0 tx=0).stop 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 msgr2=0x7f3f4406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f4406a4b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402c0b0 tx=0x7f3f340323b0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 shutdown_connections 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f44102170 0x7f3f4419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f441017b0 0x7f3f4419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 --2- 172.21.15.145:0/2689893146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f441005c0 0x7f3f4406a4b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 >> 172.21.15.145:0/2689893146 conn(0x7f3f440fbb30 msgr2=0x7f3f44104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 shutdown_connections 2024-03-28T18:37:36.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.748+0000 7f3f48bdf640 1 -- 172.21.15.145:0/2689893146 wait complete. 2024-03-28T18:37:36.761 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:36 smithi145 ceph-mon[32151]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:36.761 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:36 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1966737706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:36.761 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:36 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1391187891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:36.762 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:36 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2025147964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:36.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:37:36.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 -- 172.21.15.145:0/1694034812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c102150 msgr2=0x7f2b3c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 --2- 172.21.15.145:0/1694034812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c102150 0x7f2b3c1089e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009900 tx=0x7f2b2c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 -- 172.21.15.145:0/1694034812 shutdown_connections 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 --2- 172.21.15.145:0/1694034812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c102150 0x7f2b3c1089e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 --2- 172.21.15.145:0/1694034812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c101790 0x7f2b3c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 --2- 172.21.15.145:0/1694034812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c1005a0 0x7f2b3c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 -- 172.21.15.145:0/1694034812 >> 172.21.15.145:0/1694034812 conn(0x7f2b3c0fbb30 msgr2=0x7f2b3c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 -- 172.21.15.145:0/1694034812 shutdown_connections 2024-03-28T18:37:36.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.868+0000 7f2b40922640 1 -- 172.21.15.145:0/1694034812 wait complete. 2024-03-28T18:37:36.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 Processor -- start 2024-03-28T18:37:36.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 -- start start 2024-03-28T18:37:36.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c1005a0 0x7f2b3c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c102150 0x7f2b3c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c10c230 con 0x7f2b3c101790 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c10c0b0 con 0x7f2b3c1005a0 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b40922640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c10c3b0 con 0x7f2b3c102150 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b39d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b39d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59712/0 (socket says 172.21.15.145:59712) 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b39d9b640 1 -- 172.21.15.145:0/3429064755 learned_addr learned my addr 172.21.15.145:0/3429064755 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.869+0000 7f2b3a59c640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c1005a0 0x7f2b3c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b3ad9d640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c102150 0x7f2b3c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 -- 172.21.15.145:0/3429064755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c102150 msgr2=0x7f2b3c19de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c102150 0x7f2b3c19de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 -- 172.21.15.145:0/3429064755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c1005a0 msgr2=0x7f2b3c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c1005a0 0x7f2b3c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 -- 172.21.15.145:0/3429064755 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c19e480 con 0x7f2b3c101790 2024-03-28T18:37:36.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b39d9b640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000ecd0 tx=0x7f2b3000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3001c070 con 0x7f2b3c101790 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c068860 con 0x7f2b3c101790 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b300178f0 con 0x7f2b3c101790 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c068e00 con 0x7f2b3c101790 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.870+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30016960 con 0x7f2b3c101790 2024-03-28T18:37:36.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.871+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c107330 con 0x7f2b3c101790 2024-03-28T18:37:36.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.872+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2b3001f070 con 0x7f2b3c101790 2024-03-28T18:37:36.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.873+0000 7f2b2b7fe640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 0x7f2b240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:36.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.873+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2b300d0c60 con 0x7f2b3c101790 2024-03-28T18:37:36.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.873+0000 7f2b3a59c640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 0x7f2b240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:36.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.873+0000 7f2b3a59c640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 0x7f2b240a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c1015f0 tx=0x7f2b34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:36.878 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:36.877+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b3009d350 con 0x7f2b3c101790 2024-03-28T18:37:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:36 smithi192 ceph-mon[36589]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:36 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1966737706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:36 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1391187891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:36 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2025147964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:36.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:36 smithi164 ceph-mon[36248]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:37:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:36 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1966737706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:36 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1391187891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:36 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2025147964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:37.083 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.083+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b3c107500 con 0x7f2b3c101790 2024-03-28T18:37:37.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.083+0000 7f2b2b7fe640 1 -- 172.21.15.145:0/3429064755 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b300a13f0 con 0x7f2b3c101790 2024-03-28T18:37:37.084 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 msgr2=0x7f2b240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 0x7f2b240a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c1015f0 tx=0x7f2b34005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 msgr2=0x7f2b3c196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000ecd0 tx=0x7f2b3000b690 comp rx=0 tx=0).stop 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 shutdown_connections 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f2b240a2800 0x7f2b240a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2b3c102150 0x7f2b3c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b3c101790 0x7f2b3c196da0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 --2- 172.21.15.145:0/3429064755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b3c1005a0 0x7f2b3c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 >> 172.21.15.145:0/3429064755 conn(0x7f2b3c0fbb30 msgr2=0x7f2b3c104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 shutdown_connections 2024-03-28T18:37:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.085+0000 7f2b40922640 1 -- 172.21.15.145:0/3429064755 wait complete. 2024-03-28T18:37:37.097 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:37:37.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 -- 172.21.15.145:0/3226242667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8107380 msgr2=0x7fc8a81097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/3226242667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8107380 0x7fc8a81097d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009ca0 tx=0x7fc8a002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 -- 172.21.15.145:0/3226242667 shutdown_connections 2024-03-28T18:37:37.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/3226242667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8107380 0x7fc8a81097d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.203 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/3226242667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8104a50 0x7fc8a8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/3226242667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8102120 0x7fc8a8104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 -- 172.21.15.145:0/3226242667 >> 172.21.15.145:0/3226242667 conn(0x7fc8a80fbb30 msgr2=0x7fc8a80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 -- 172.21.15.145:0/3226242667 shutdown_connections 2024-03-28T18:37:37.204 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.202+0000 7fc8acdc6640 1 -- 172.21.15.145:0/3226242667 wait complete. 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 Processor -- start 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 -- start start 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 0x7fc8a81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8107380 0x7fc8a819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a810d340 con 0x7fc8a8104a50 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a810d1c0 con 0x7fc8a8102120 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8acdc6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a810d4c0 con 0x7fc8a8107380 2024-03-28T18:37:37.206 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59738/0 (socket says 172.21.15.145:59738) 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 0x7fc8a81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a5d9b640 1 -- 172.21.15.145:0/1264139962 learned_addr learned my addr 172.21.15.145:0/1264139962 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a6d9d640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8107380 0x7fc8a819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a5d9b640 1 -- 172.21.15.145:0/1264139962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8107380 msgr2=0x7fc8a819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.203+0000 7fc8a5d9b640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8107380 0x7fc8a819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8a5d9b640 1 -- 172.21.15.145:0/1264139962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 msgr2=0x7fc8a81979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8a5d9b640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 0x7fc8a81979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8a5d9b640 1 -- 172.21.15.145:0/1264139962 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a819f5d0 con 0x7fc8a8104a50 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8a659c640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 0x7fc8a81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8a5d9b640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00e9c0 tx=0x7fc89c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c00cd80 con 0x7fc8a8104a50 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc89c005500 con 0x7fc8a8104a50 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c010690 con 0x7fc8a8104a50 2024-03-28T18:37:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a819f8c0 con 0x7fc8a8104a50 2024-03-28T18:37:37.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.204+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a819fe60 con 0x7fc8a8104a50 2024-03-28T18:37:37.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.205+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a80fd620 con 0x7fc8a8104a50 2024-03-28T18:37:37.208 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.207+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc89c0046c0 con 0x7fc8a8104a50 2024-03-28T18:37:37.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.207+0000 7fc8977fe640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 0x7fc8900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.207+0000 7fc8a659c640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 0x7fc8900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.207+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc89c0cf250 con 0x7fc8a8104a50 2024-03-28T18:37:37.209 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.208+0000 7fc8a659c640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 0x7fc8900a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc898009fc0 tx=0x7fc898005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.212 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.212+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc89c053740 con 0x7fc8a8104a50 2024-03-28T18:37:37.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.390+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc8a802d280 con 0x7fc8a8104a50 2024-03-28T18:37:37.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.391+0000 7fc8977fe640 1 -- 172.21.15.145:0/1264139962 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc89c09bf90 con 0x7fc8a8104a50 2024-03-28T18:37:37.392 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 msgr2=0x7fc8900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 0x7fc8900a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc898009fc0 tx=0x7fc898005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 msgr2=0x7fc8a8197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00e9c0 tx=0x7fc89c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 shutdown_connections 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc8900a2880 0x7fc8900a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8a8107380 0x7fc8a819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8a8104a50 0x7fc8a8197ef0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 --2- 172.21.15.145:0/1264139962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8a8102120 0x7fc8a81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 >> 172.21.15.145:0/1264139962 conn(0x7fc8a80fbb30 msgr2=0x7fc8a8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 shutdown_connections 2024-03-28T18:37:37.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.393+0000 7fc8acdc6640 1 -- 172.21.15.145:0/1264139962 wait complete. 2024-03-28T18:37:37.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 -- 172.21.15.145:0/723302321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e81021e0 msgr2=0x7fc1e81045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/723302321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e81021e0 0x7fc1e81045d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009860 tx=0x7fc1d802bb70 comp rx=0 tx=0).stop 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 -- 172.21.15.145:0/723302321 shutdown_connections 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/723302321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8107440 0x7fc1e8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/723302321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e8104b10 0x7fc1e8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/723302321 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e81021e0 0x7fc1e81045d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 -- 172.21.15.145:0/723302321 >> 172.21.15.145:0/723302321 conn(0x7fc1e80fbb30 msgr2=0x7fc1e80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 -- 172.21.15.145:0/723302321 shutdown_connections 2024-03-28T18:37:37.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.512+0000 7fc1edb7e640 1 -- 172.21.15.145:0/723302321 wait complete. 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 Processor -- start 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 -- start start 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8104b10 0x7fc1e8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e8107440 0x7fc1e819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e810d330 con 0x7fc1e81021e0 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e810d1b0 con 0x7fc1e8104b10 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1edb7e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e810d4b0 con 0x7fc1e8107440 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59758/0 (socket says 172.21.15.145:59758) 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 -- 172.21.15.145:0/3228038266 learned_addr learned my addr 172.21.15.145:0/3228038266 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e7fff640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e8107440 0x7fc1e819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e6ffd640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8104b10 0x7fc1e8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 -- 172.21.15.145:0/3228038266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e8107440 msgr2=0x7fc1e819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.516 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e8107440 0x7fc1e819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 -- 172.21.15.145:0/3228038266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8104b10 msgr2=0x7fc1e8197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8104b10 0x7fc1e8197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.513+0000 7fc1e77fe640 1 -- 172.21.15.145:0/3228038266 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e819f650 con 0x7fc1e81021e0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1e77fe640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8005940 tx=0x7fc1d8032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8041020 con 0x7fc1e81021e0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e819f8e0 con 0x7fc1e81021e0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1d802c9f0 con 0x7fc1e81021e0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8039040 con 0x7fc1e81021e0 2024-03-28T18:37:37.517 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.514+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e819fdd0 con 0x7fc1e81021e0 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.515+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e80fd620 con 0x7fc1e81021e0 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.516+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc1d8044020 con 0x7fc1e81021e0 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.517+0000 7fc1e4ff9640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 0x7fc1d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.517+0000 7fc1e6ffd640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 0x7fc1d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.517+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc1d80f0710 con 0x7fc1e81021e0 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.517+0000 7fc1e6ffd640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 0x7fc1d00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8198f40 tx=0x7fc1dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.521+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1d804b020 con 0x7fc1e81021e0 2024-03-28T18:37:37.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.702+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc1e80622a0 con 0x7fc1e81021e0 2024-03-28T18:37:37.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.702+0000 7fc1e4ff9640 1 -- 172.21.15.145:0/3228038266 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc1d80bd450 con 0x7fc1e81021e0 2024-03-28T18:37:37.703 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 msgr2=0x7fc1d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 0x7fc1d00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8198f40 tx=0x7fc1dc009260 comp rx=0 tx=0).stop 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 msgr2=0x7fc1e8197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8005940 tx=0x7fc1d8032470 comp rx=0 tx=0).stop 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 shutdown_connections 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc1d00a2800 0x7fc1d00a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1e8107440 0x7fc1e819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1e8104b10 0x7fc1e8197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 --2- 172.21.15.145:0/3228038266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1e81021e0 0x7fc1e8197a30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 >> 172.21.15.145:0/3228038266 conn(0x7fc1e80fbb30 msgr2=0x7fc1e8102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 shutdown_connections 2024-03-28T18:37:37.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.706+0000 7fc1edb7e640 1 -- 172.21.15.145:0/3228038266 wait complete. 2024-03-28T18:37:37.716 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2689893146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.716 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3429064755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.716 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1264139962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 -- 172.21.15.145:0/3981316105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c102200 msgr2=0x7f932c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 --2- 172.21.15.145:0/3981316105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c102200 0x7f932c1045f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f931c009900 tx=0x7f931c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 -- 172.21.15.145:0/3981316105 shutdown_connections 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 --2- 172.21.15.145:0/3981316105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c107460 0x7f932c109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 --2- 172.21.15.145:0/3981316105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c104b30 0x7f932c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.820 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 --2- 172.21.15.145:0/3981316105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c102200 0x7f932c1045f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.821 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 -- 172.21.15.145:0/3981316105 >> 172.21.15.145:0/3981316105 conn(0x7f932c0fbb30 msgr2=0x7f932c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.821 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 -- 172.21.15.145:0/3981316105 shutdown_connections 2024-03-28T18:37:37.821 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.819+0000 7f9331680640 1 -- 172.21.15.145:0/3981316105 wait complete. 2024-03-28T18:37:37.822 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 Processor -- start 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 -- start start 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c102200 0x7f932c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c104b30 0x7f932c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10d390 con 0x7f932c107460 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10d210 con 0x7f932c102200 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9331680640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10d510 con 0x7f932c104b30 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f932b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f932b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59766/0 (socket says 172.21.15.145:59766) 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f932b7fe640 1 -- 172.21.15.145:0/25734960 learned_addr learned my addr 172.21.15.145:0/25734960 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f932affd640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c102200 0x7f932c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.820+0000 7f9323fff640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c104b30 0x7f932c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.823 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 -- 172.21.15.145:0/25734960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c104b30 msgr2=0x7f932c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c104b30 0x7f932c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 -- 172.21.15.145:0/25734960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c102200 msgr2=0x7f932c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c102200 0x7f932c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 -- 172.21.15.145:0/25734960 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c19f600 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f932b7fe640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f932400eb10 tx=0x7f932400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324004b20 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c19f8f0 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9324004cc0 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324010760 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.821+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c19fdf0 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.822+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c0fd620 con 0x7f932c107460 2024-03-28T18:37:37.824 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.824+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9324005000 con 0x7f932c107460 2024-03-28T18:37:37.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.824+0000 7f9328ff9640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 0x7f93180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.825+0000 7f932affd640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 0x7f93180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.825+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f93240cf560 con 0x7f932c107460 2024-03-28T18:37:37.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.825+0000 7f932affd640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 0x7f93180a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f931c02c240 tx=0x7f931c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:37.828+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9324053740 con 0x7f932c107460 2024-03-28T18:37:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2689893146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3429064755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1264139962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2689893146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3429064755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:37:37.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1264139962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:37:38.048 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.047+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f932c02d070 con 0x7f932c107460 2024-03-28T18:37:38.049 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.048+0000 7f9328ff9640 1 -- 172.21.15.145:0/25734960 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7f932409c2a0 con 0x7f932c107460 2024-03-28T18:37:38.049 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:38.049 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:37:38.049 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:37:38.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 msgr2=0x7f93180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 0x7f93180a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f931c02c240 tx=0x7f931c02df90 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 msgr2=0x7f932c19ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f932400eb10 tx=0x7f932400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 shutdown_connections 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c107460 0x7f932c19ef90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f93180a2800 0x7f93180a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f932c104b30 0x7f932c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 --2- 172.21.15.145:0/25734960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f932c102200 0x7f932c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.050+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 >> 172.21.15.145:0/25734960 conn(0x7f932c0fbb30 msgr2=0x7f932c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.051+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 shutdown_connections 2024-03-28T18:37:38.052 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.051+0000 7f9331680640 1 -- 172.21.15.145:0/25734960 wait complete. 2024-03-28T18:37:38.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:37:38.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.165+0000 7f0778737640 1 -- 172.21.15.145:0/3358549973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0770103290 msgr2=0x7f0770109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.165+0000 7f0778737640 1 --2- 172.21.15.145:0/3358549973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0770103290 0x7f0770109b20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0768009be0 tx=0x7f076802bef0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 -- 172.21.15.145:0/3358549973 shutdown_connections 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 --2- 172.21.15.145:0/3358549973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0770103290 0x7f0770109b20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 --2- 172.21.15.145:0/3358549973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07701028d0 0x7f0770102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 --2- 172.21.15.145:0/3358549973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701016e0 0x7f0770101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 -- 172.21.15.145:0/3358549973 >> 172.21.15.145:0/3358549973 conn(0x7f07700fcc50 msgr2=0x7f07700ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 -- 172.21.15.145:0/3358549973 shutdown_connections 2024-03-28T18:37:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 -- 172.21.15.145:0/3358549973 wait complete. 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 Processor -- start 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.166+0000 7f0778737640 1 -- start start 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 0x7f0770197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 0x7f0770197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077010d360 con 0x7f0770103290 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077010d1e0 con 0x7f07701028d0 2024-03-28T18:37:38.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077010d4e0 con 0x7f07701016e0 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59790/0 (socket says 172.21.15.145:59790) 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f07764d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 0x7f0770197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 -- 172.21.15.145:0/932789294 learned_addr learned my addr 172.21.15.145:0/932789294 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0775cd2640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 0x7f0770197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 -- 172.21.15.145:0/932789294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 msgr2=0x7f0770197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 0x7f0770197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 -- 172.21.15.145:0/932789294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 msgr2=0x7f0770197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 0x7f0770197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 -- 172.21.15.145:0/932789294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077019f5a0 con 0x7f0770103290 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0775cd2640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 0x7f0770197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f07764d3640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 0x7f0770197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0776cd4640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0768036040 tx=0x7f07680337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768043020 con 0x7f0770103290 2024-03-28T18:37:38.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077019f830 con 0x7f0770103290 2024-03-28T18:37:38.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f076802ca60 con 0x7f0770103290 2024-03-28T18:37:38.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0768034700 con 0x7f0770103290 2024-03-28T18:37:38.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.167+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077019fd80 con 0x7f0770103290 2024-03-28T18:37:38.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.168+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0770108a00 con 0x7f0770103290 2024-03-28T18:37:38.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.170+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0768046020 con 0x7f0770103290 2024-03-28T18:37:38.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.170+0000 7f07677fe640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 0x7f075c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.170+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f07680f1bb0 con 0x7f0770103290 2024-03-28T18:37:38.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.170+0000 7f07764d3640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 0x7f075c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.171+0000 7f07764d3640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 0x7f075c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0770102730 tx=0x7f076c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.174 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.174+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f076804d020 con 0x7f0770103290 2024-03-28T18:37:38.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.356+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07700622a0 con 0x7f0770103290 2024-03-28T18:37:38.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.357+0000 7f07677fe640 1 -- 172.21.15.145:0/932789294 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24703 (secure 0 0 0) 0x7f07680be8f0 con 0x7f0770103290 2024-03-28T18:37:38.357 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:38.358 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":93,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:36:54.585370+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 msgr2=0x7f075c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 0x7f075c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0770102730 tx=0x7f076c005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 msgr2=0x7f077019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0768036040 tx=0x7f07680337b0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 shutdown_connections 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0770103290 0x7f077019ef30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07701028d0 0x7f0770197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f075c0a2800 0x7f075c0a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 --2- 172.21.15.145:0/932789294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07701016e0 0x7f0770197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 >> 172.21.15.145:0/932789294 conn(0x7f07700fcc50 msgr2=0x7f0770105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 shutdown_connections 2024-03-28T18:37:38.361 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.360+0000 7f0778737640 1 -- 172.21.15.145:0/932789294 wait complete. 2024-03-28T18:37:38.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:37:38.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 -- 172.21.15.145:0/2396215087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901016f0 msgr2=0x7f6290101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 --2- 172.21.15.145:0/2396215087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901016f0 0x7f6290101b10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6280009900 tx=0x7f628002bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:38.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 -- 172.21.15.145:0/2396215087 shutdown_connections 2024-03-28T18:37:38.474 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 --2- 172.21.15.145:0/2396215087 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901032a0 0x7f6290109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 --2- 172.21.15.145:0/2396215087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901028e0 0x7f6290102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 --2- 172.21.15.145:0/2396215087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901016f0 0x7f6290101b10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 -- 172.21.15.145:0/2396215087 >> 172.21.15.145:0/2396215087 conn(0x7f62900fcc40 msgr2=0x7f62900ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 -- 172.21.15.145:0/2396215087 shutdown_connections 2024-03-28T18:37:38.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.474+0000 7f6295728640 1 -- 172.21.15.145:0/2396215087 wait complete. 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 Processor -- start 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 -- start start 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901016f0 0x7f6290197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901032a0 0x7f629019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010d340 con 0x7f62901028e0 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010d1c0 con 0x7f62901032a0 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f6295728640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010d4c0 con 0x7f62901016f0 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f628e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f628e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59808/0 (socket says 172.21.15.145:59808) 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f628e7fc640 1 -- 172.21.15.145:0/3423987891 learned_addr learned my addr 172.21.15.145:0/3423987891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.475+0000 7f628effd640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901016f0 0x7f6290197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628f7fe640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901032a0 0x7f629019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 -- 172.21.15.145:0/3423987891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901016f0 msgr2=0x7f6290197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901016f0 0x7f6290197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 -- 172.21.15.145:0/3423987891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901032a0 msgr2=0x7f629019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901032a0 0x7f629019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 -- 172.21.15.145:0/3423987891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629019f660 con 0x7f62901028e0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f628e7fc640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f627800ea60 tx=0x7f627800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627800cd20 con 0x7f62901028e0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629019f950 con 0x7f62901028e0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62780035f0 con 0x7f62901028e0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278010690 con 0x7f62901028e0 2024-03-28T18:37:38.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.476+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629019fea0 con 0x7f62901028e0 2024-03-28T18:37:38.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.477+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6290108a60 con 0x7f62901028e0 2024-03-28T18:37:38.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.478+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f62780037a0 con 0x7f62901028e0 2024-03-28T18:37:38.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.479+0000 7f6287fff640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 0x7f627c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.479+0000 7f628effd640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 0x7f627c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.479+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f62780cf0d0 con 0x7f62901028e0 2024-03-28T18:37:38.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.479+0000 7f628effd640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 0x7f627c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.483+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f627809be10 con 0x7f62901028e0 2024-03-28T18:37:38.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.720+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62900622a0 con 0x7f62901028e0 2024-03-28T18:37:38.721 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.721+0000 7f6287fff640 1 -- 172.21.15.145:0/3423987891 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7f627809b710 con 0x7f62901028e0 2024-03-28T18:37:38.721 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:38.721 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:37:38.721 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 msgr2=0x7f627c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 0x7f627c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280005fd0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 msgr2=0x7f6290197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f627800ea60 tx=0x7f627800ee60 comp rx=0 tx=0).stop 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 shutdown_connections 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62901032a0 0x7f629019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62901028e0 0x7f6290197f80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.723 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f627c0a2800 0x7f627c0a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.724 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 --2- 172.21.15.145:0/3423987891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62901016f0 0x7f6290197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.724 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 >> 172.21.15.145:0/3423987891 conn(0x7f62900fcc40 msgr2=0x7f6290105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.724 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 shutdown_connections 2024-03-28T18:37:38.724 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.723+0000 7f6295728640 1 -- 172.21.15.145:0/3423987891 wait complete. 2024-03-28T18:37:38.733 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:38 smithi145 ceph-mon[32151]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:38.733 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:38 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3228038266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:38.733 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:38 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/25734960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.733 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:38 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/932789294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:35:50.266038+0000', 'modified': '2024-03-28T18:36:13.026843+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24496, 'mds_1': 24491, 'mds_2': 24469, 'mds_3': 24461, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.145:6839/1703430127', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6838', 'nonce': 1703430127}, {'type': 'v1', 'addr': '172.21.15.145:6839', 'nonce': 1703430127}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.145:6841/2292179731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6840', 'nonce': 2292179731}, {'type': 'v1', 'addr': '172.21.15.145:6841', 'nonce': 2292179731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6833/3835667312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6832', 'nonce': 3835667312}, {'type': 'v1', 'addr': '172.21.15.164:6833', 'nonce': 3835667312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6833/3704007459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 3704007459}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 3704007459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6835/3793155920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 3793155920}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 3793155920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/1226474966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 1226474966}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 1226474966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.164:6837/1634173775', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 1634173775}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 1634173775}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.192:6837/2153191464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 2153191464}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 2153191464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24491': {'gid': 24491, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.164:6839/644460294', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6838', 'nonce': 644460294}, {'type': 'v1', 'addr': '172.21.15.164:6839', 'nonce': 644460294}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.192:6839/2516430938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 2516430938}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 2516430938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24496, 'qdb_cluster': [24469, 24491, 14631, 24461, 24496]} 2024-03-28T18:37:38.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T18:37:38.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- 172.21.15.145:0/2277853014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 msgr2=0x7f33841089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2277853014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f33841089e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f337c009be0 tx=0x7f337c02bef0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- 172.21.15.145:0/2277853014 shutdown_connections 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2277853014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f33841089e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2277853014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3384101790 0x7f3384101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2277853014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33841005a0 0x7f33841009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- 172.21.15.145:0/2277853014 >> 172.21.15.145:0/2277853014 conn(0x7f33840fbb30 msgr2=0x7f33840fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- 172.21.15.145:0/2277853014 shutdown_connections 2024-03-28T18:37:38.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- 172.21.15.145:0/2277853014 wait complete. 2024-03-28T18:37:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 Processor -- start 2024-03-28T18:37:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.840+0000 7f338b7f3640 1 -- start start 2024-03-28T18:37:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33841005a0 0x7f3384197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f338419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410c250 con 0x7f3384101790 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410c0d0 con 0x7f3384102150 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338410c3d0 con 0x7f33841005a0 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59818/0 (socket says 172.21.15.145:59818) 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 -- 172.21.15.145:0/2244521764 learned_addr learned my addr 172.21.15.145:0/2244521764 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338958f640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33841005a0 0x7f3384197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3389d90640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f338419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 -- 172.21.15.145:0/2244521764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33841005a0 msgr2=0x7f3384197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33841005a0 0x7f3384197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 -- 172.21.15.145:0/2244521764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 msgr2=0x7f338419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f338419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 -- 172.21.15.145:0/2244521764 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338419f5a0 con 0x7f3384101790 2024-03-28T18:37:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f3388d8e640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f337400ea60 tx=0x7f337400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.844 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f337a7fc640 1 -- 172.21.15.145:0/2244521764 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337400cde0 con 0x7f3384101790 2024-03-28T18:37:38.844 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338419f890 con 0x7f3384101790 2024-03-28T18:37:38.844 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f337a7fc640 1 -- 172.21.15.145:0/2244521764 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33740035f0 con 0x7f3384101790 2024-03-28T18:37:38.844 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f337a7fc640 1 -- 172.21.15.145:0/2244521764 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374010670 con 0x7f3384101790 2024-03-28T18:37:38.844 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.841+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338419fde0 con 0x7f3384101790 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f337a7fc640 1 -- 172.21.15.145:0/2244521764 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f33740037a0 con 0x7f3384101790 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 Processor -- start 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 -- start start 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f337a7fc640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 0x7f33700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79d93b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.850 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d9b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79cf8c20 con 0x563c79d94130 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79ca1a40 con 0x563c79d992a0 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.844+0000 7f338e3da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79cf4cc0 con 0x563c79d93740 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3389d90640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 0x7f33700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f337a7fc640 1 -- 172.21.15.145:0/2244521764 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f33740cf0b0 con 0x7f3384101790 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3389d90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d9b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f338958f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79d93b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59826/0 (socket says 172.21.15.145:59826) 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 -- 172.21.15.145:0/1051084 learned_addr learned my addr 172.21.15.145:0/1051084 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 -- 172.21.15.145:0/1051084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 msgr2=0x563c79d93b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79d93b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.851 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 -- 172.21.15.145:0/1051084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 msgr2=0x563c79d9b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d9b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 -- 172.21.15.145:0/1051084 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c79cc3f90 con 0x563c79d94130 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f338958f640 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79d93b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3389d90640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 0x7f33700a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f337c006010 tx=0x7f337c009840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f337400d010 tx=0x7f337409b230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec7b944598d72c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f336f7fe640 1 -- 172.21.15.145:0/1051084 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337409bdf0 con 0x563c79d94130 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f336f7fe640 1 -- 172.21.15.145:0/1051084 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f337409b6f0 con 0x563c79d94130 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.845+0000 7f336f7fe640 1 -- 172.21.15.145:0/1051084 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374097d60 con 0x563c79d94130 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- 172.21.15.145:0/1051084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 msgr2=0x563c79d98c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f337400d010 tx=0x7f337409b230 comp rx=0 tx=0).stop 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- 172.21.15.145:0/1051084 shutdown_connections 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d9b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79d98c30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 --2- 172.21.15.145:0/1051084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79d93b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- 172.21.15.145:0/1051084 >> 172.21.15.145:0/1051084 conn(0x563c79d5ba60 msgr2=0x563c79d8f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- 172.21.15.145:0/1051084 shutdown_connections 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- 172.21.15.145:0/1051084 wait complete. 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 -- start start 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.846+0000 7f338e3da740 1 Processor -- start 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79d9e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79d9c250 con 0x563c79d93740 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79cb4e90 con 0x563c79d992a0 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79d9c3d0 con 0x563c79d94130 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338e3da740 10 client.0 _ll_register_callbacks cb 0x563c79a0cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59838/0 (socket says 172.21.15.145:59838) 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 -- 172.21.15.145:0/4002965666 learned_addr learned my addr 172.21.15.145:0/4002965666 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f3388d8e640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79d9e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f3389d90640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 -- 172.21.15.145:0/4002965666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 msgr2=0x563c79d9e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79d9e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 -- 172.21.15.145:0/4002965666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 msgr2=0x563c79da51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 -- 172.21.15.145:0/4002965666 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c79cc19d0 con 0x563c79d93740 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f3388d8e640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79d9e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f338958f640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f33800116a0 tx=0x7f3380011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a6344944cd730d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f336cff9640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.847+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338001e020 con 0x563c79d93740 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f337409b6f0 con 0x563c79d93740 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f338000fb90 con 0x563c79d93740 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3380026050 con 0x563c79d93740 2024-03-28T18:37:38.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 10 client.15225 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563c79a0d2d0 con 0x563c79d93740 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f337409b6f0 con 0x563c79d93740 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3380019070 con 0x563c79d93740 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 10 client.15225 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 10 client.15225 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 20 client.15225 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79daa4c0 0x563c79da9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 4 client.15225 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 --> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c79d06160 con 0x563c79daa4c0 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.848+0000 7f3388d8e640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79daa4c0 0x563c79da9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f336cff9640 10 client.15225 ms_handle_connect on v2:172.21.15.145:6840/2292179731 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f3388d8e640 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79daa4c0 0x563c79da9540 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d7d5871927de87c1 server_cookie=4c5c4ce3ce07f92f in_seq=0 out_seq=0 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f336cff9640 1 -- 172.21.15.145:0/4002965666 <== mds.4 v2:172.21.15.145:6840/2292179731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c79d06160 con 0x563c79daa4c0 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f336cff9640 10 client.15225 handle_command_reply: tid=0 2024-03-28T18:37:38.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 client.15225 shutdown 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 msgr2=0x563c79d9dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f33800116a0 tx=0x7f3380011b60 comp rx=0 tx=0).stop 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 shutdown_connections 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79daa4c0 0x563c79da9540 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79d9e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f336cff9640 0 client.15225 ms_handle_reset on v2:172.21.15.145:6840/2292179731 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 --2- 172.21.15.145:0/4002965666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d9dc20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.849+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 >> 172.21.15.145:0/4002965666 conn(0x563c79d5ba60 msgr2=0x563c79d8e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.850+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 shutdown_connections 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.850+0000 7f338e3da740 1 -- 172.21.15.145:0/4002965666 wait complete. 2024-03-28T18:37:38.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.850+0000 7f338e3da740 20 client.15225 trim_cache size 0 max 16384 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 Processor -- start 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 -- start start 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79da50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79da9ce0 con 0x563c79d93740 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79daa340 con 0x563c79d992a0 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338e3da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79daa9a0 con 0x563c79d94130 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338958f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338958f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59842/0 (socket says 172.21.15.145:59842) 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f338958f640 1 -- 172.21.15.145:0/1131229528 learned_addr learned my addr 172.21.15.145:0/1131229528 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79da50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.866+0000 7f3389d90640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 -- 172.21.15.145:0/1131229528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 msgr2=0x563c79d922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d922f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 -- 172.21.15.145:0/1131229528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 msgr2=0x563c79d8f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 -- 172.21.15.145:0/1131229528 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c79ad76b0 con 0x563c79d94130 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f338958f640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79da50c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x563c79d94530 tx=0x7f33740999f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3dacd47544e6761 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f336effd640 1 -- 172.21.15.145:0/1131229528 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337409b6f0 con 0x563c79d94130 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f336effd640 1 -- 172.21.15.145:0/1131229528 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3374084070 con 0x563c79d94130 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.867+0000 7f336effd640 1 -- 172.21.15.145:0/1131229528 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374085360 con 0x563c79d94130 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- 172.21.15.145:0/1131229528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 msgr2=0x563c79da50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79da50c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x563c79d94530 tx=0x7f33740999f0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- 172.21.15.145:0/1131229528 shutdown_connections 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79d922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.873 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d94130 0x563c79da50c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- 172.21.15.145:0/1131229528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d93740 0x563c79d8f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- 172.21.15.145:0/1131229528 >> 172.21.15.145:0/1131229528 conn(0x563c79d5ba60 msgr2=0x563c79d8ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- 172.21.15.145:0/1131229528 shutdown_connections 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- 172.21.15.145:0/1131229528 wait complete. 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- start start 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 Processor -- start 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79dab660 con 0x563c79d94130 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79dab000 con 0x563c79d992a0 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.868+0000 7f338e3da740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c79dabcc0 con 0x563c79d93740 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338e3da740 10 client.0 _ll_register_callbacks cb 0x7f3380009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.874 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52360/0 (socket says 172.21.15.145:52360) 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 -- 172.21.15.145:0/1008812054 learned_addr learned my addr 172.21.15.145:0/1008812054 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f3388d8e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59856/0 (socket says 172.21.15.145:59856) 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f3389d90640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 -- 172.21.15.145:0/1008812054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 msgr2=0x563c79da3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 -- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 msgr2=0x563c79da0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 -- 172.21.15.145:0/1008812054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c79da4500 con 0x563c79d93740 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338958f640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3380019460 tx=0x7f3380005a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d333950c7270b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f336cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380030030 con 0x563c79d93740 2024-03-28T18:37:38.875 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3374084070 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3380017050 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f338003a050 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338e3da740 10 client.25090 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.869+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563c79e05650 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3374084070 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3380019040 con 0x563c79d93740 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 10 client.25090 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 10 client.25090 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 20 client.25090 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79e09cd0 0x563c79e05880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 4 client.25090 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 --> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563c79d004b0 con 0x563c79e09cd0 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79e09cd0 0x563c79e05880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f336cff9640 10 client.25090 ms_handle_connect on v2:172.21.15.145:6840/2292179731 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f3388d8e640 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79e09cd0 0x563c79e05880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=502c119d39c1c737 server_cookie=98734caf91946126 in_seq=0 out_seq=0 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f336cff9640 1 -- 172.21.15.145:0/1008812054 <== mds.4 v2:172.21.15.145:6840/2292179731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x563c79d004b0 con 0x563c79e09cd0 2024-03-28T18:37:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.870+0000 7f336cff9640 10 client.25090 handle_command_reply: tid=0 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 client.25090 shutdown 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 msgr2=0x563c79da09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3380019460 tx=0x7f3380005a70 comp rx=0 tx=0).stop 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 shutdown_connections 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x563c79e09cd0 0x563c79e05880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563c79d992a0 0x563c79da3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f336cff9640 0 client.25090 ms_handle_reset on v2:172.21.15.145:6840/2292179731 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563c79d94130 0x563c79da0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 --2- 172.21.15.145:0/1008812054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563c79d93740 0x563c79da09b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 >> 172.21.15.145:0/1008812054 conn(0x563c79d5ba60 msgr2=0x563c79d8ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 shutdown_connections 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 1 -- 172.21.15.145:0/1008812054 wait complete. 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338e3da740 20 client.25090 trim_cache size 0 max 16384 2024-03-28T18:37:38.877 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 4, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 14631, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 86.864064782, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 93, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 117.52776253099999 2024-03-28T18:37:38.878 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 msgr2=0x7f33700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 0x7f33700a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f337c006010 tx=0x7f337c009840 comp rx=0 tx=0).stop 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 msgr2=0x7f3384197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f337400ea60 tx=0x7f337400ef20 comp rx=0 tx=0).stop 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.871+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 shutdown_connections 2024-03-28T18:37:38.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3384102150 0x7f338419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3384101790 0x7f3384197ec0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f33700a2800 0x7f33700a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 --2- 172.21.15.145:0/2244521764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f33841005a0 0x7f3384197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 >> 172.21.15.145:0/2244521764 conn(0x7f33840fbb30 msgr2=0x7f33840fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 shutdown_connections 2024-03-28T18:37:38.880 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.872+0000 7f338b7f3640 1 -- 172.21.15.145:0/2244521764 wait complete. 2024-03-28T18:37:38.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T18:37:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:38 smithi192 ceph-mon[36589]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:38 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3228038266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:38 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/25734960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:38 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/932789294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:38 smithi164 ceph-mon[36248]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T18:37:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:38 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3228038266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:37:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:38 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/25734960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:38 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/932789294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:37:38.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.986+0000 7fa258285640 1 -- 172.21.15.145:0/3478159853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2501005a0 msgr2=0x7fa2501009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.986+0000 7fa258285640 1 --2- 172.21.15.145:0/3478159853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2501005a0 0x7fa2501009c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009900 tx=0x7fa24c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:38.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- 172.21.15.145:0/3478159853 shutdown_connections 2024-03-28T18:37:38.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- 172.21.15.145:0/3478159853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa250102150 0x7fa2501089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- 172.21.15.145:0/3478159853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa250101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- 172.21.15.145:0/3478159853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2501005a0 0x7fa2501009c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- 172.21.15.145:0/3478159853 >> 172.21.15.145:0/3478159853 conn(0x7fa2500fbb30 msgr2=0x7fa2500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- 172.21.15.145:0/3478159853 shutdown_connections 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- 172.21.15.145:0/3478159853 wait complete. 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 Processor -- start 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- start start 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa25019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010c200 con 0x7fa250102150 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010c080 con 0x7fa250101790 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa258285640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010c380 con 0x7fa2501005a0 2024-03-28T18:37:38.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa256822640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa256021640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa256021640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52374/0 (socket says 172.21.15.145:52374) 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa256822640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59872/0 (socket says 172.21.15.145:59872) 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.987+0000 7fa255820640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa25019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256021640 1 -- 172.21.15.145:0/129355539 learned_addr learned my addr 172.21.15.145:0/129355539 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 -- 172.21.15.145:0/129355539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 msgr2=0x7fa250069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 -- 172.21.15.145:0/129355539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 msgr2=0x7fa25019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa25019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 -- 172.21.15.145:0/129355539 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25019f680 con 0x7fa250102150 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256021640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa255820640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa25019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa256822640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa248009fd0 tx=0x7fa24800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa2477fe640 1 -- 172.21.15.145:0/129355539 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24800ce20 con 0x7fa250102150 2024-03-28T18:37:38.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa2477fe640 1 -- 172.21.15.145:0/129355539 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa248005020 con 0x7fa250102150 2024-03-28T18:37:38.991 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25019f910 con 0x7fa250102150 2024-03-28T18:37:38.991 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa2477fe640 1 -- 172.21.15.145:0/129355539 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248010720 con 0x7fa250102150 2024-03-28T18:37:38.991 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.988+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25019fe10 con 0x7fa250102150 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa2477fe640 1 -- 172.21.15.145:0/129355539 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa248005540 con 0x7fa250102150 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 Processor -- start 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 -- start start 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56589c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e5658c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e564e9c20 con 0x563e56585130 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e56492a40 con 0x563e56584740 2024-03-28T18:37:38.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa25ae79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e564e5cc0 con 0x563e5658a2a0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa256021640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56589c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51638/0 (socket says 172.21.15.145:51638) 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 -- 172.21.15.145:0/753015909 learned_addr learned my addr 172.21.15.145:0/753015909 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e5658c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 -- 172.21.15.145:0/753015909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 msgr2=0x563e5658c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e5658c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 -- 172.21.15.145:0/753015909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 msgr2=0x563e56589c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56589c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.991+0000 7fa255820640 1 -- 172.21.15.145:0/753015909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e564b4f90 con 0x563e56584740 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa2477fe640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa256021640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa255820640 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa240011be0 tx=0x7fa240009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa822f6de82433c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa2477fe640 1 -- 172.21.15.145:0/129355539 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa2480cf4d0 con 0x7fa250102150 2024-03-28T18:37:38.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa244bfe640 1 -- 172.21.15.145:0/753015909 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000fc50 con 0x563e56584740 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa244bfe640 1 -- 172.21.15.145:0/753015909 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa24000fdf0 con 0x563e56584740 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.992+0000 7fa244bfe640 1 -- 172.21.15.145:0/753015909 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000e640 con 0x563e56584740 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- 172.21.15.145:0/753015909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 msgr2=0x563e56584b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa256021640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02c8e0 tx=0x7fa24c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa240011be0 tx=0x7fa240009ba0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- 172.21.15.145:0/753015909 shutdown_connections 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e5658c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56589c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- 172.21.15.145:0/753015909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56584b60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- 172.21.15.145:0/753015909 >> 172.21.15.145:0/753015909 conn(0x563e5654ca60 msgr2=0x563e565800b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- 172.21.15.145:0/753015909 shutdown_connections 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- 172.21.15.145:0/753015909 wait complete. 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- start start 2024-03-28T18:37:38.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 Processor -- start 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56585130 0x563e5658f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e56596200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5658d250 con 0x563e5658a2a0 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e564a5e90 con 0x563e56584740 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5658d3d0 con 0x563e56585130 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa25ae79740 10 client.0 _ll_register_callbacks cb 0x563e561fdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa256021640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa256021640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51640/0 (socket says 172.21.15.145:51640) 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa256021640 1 -- 172.21.15.145:0/57381152 learned_addr learned my addr 172.21.15.145:0/57381152 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.993+0000 7fa256822640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e56596200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa255820640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56585130 0x563e5658f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 -- 172.21.15.145:0/57381152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56585130 msgr2=0x563e5658f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56585130 0x563e5658f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 -- 172.21.15.145:0/57381152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 msgr2=0x563e56596200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e56596200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 -- 172.21.15.145:0/57381152 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e564b29d0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256822640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e56596200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa256021640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02c710 tx=0x7fa24c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd608197a03a13fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa239ffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c04f070 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa24c0050d0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa24000fdf0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24c0052d0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa25ae79740 10 client.25004 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563e561fe2d0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.994+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa24000fdf0 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa24c054460 con 0x563e56584740 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 10 client.25004 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 10 client.25004 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:37:39.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 20 client.25004 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e5659b400 0x563e5659a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 4 client.25004 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 --> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e564f7160 con 0x563e5659b400 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa255820640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e5659b400 0x563e5659a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa239ffb640 10 client.25004 ms_handle_connect on v2:172.21.15.164:6836/1634173775 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.995+0000 7fa255820640 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e5659b400 0x563e5659a4e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a472b2dd3ab180c1 server_cookie=42954f5ba6c60dd8 in_seq=0 out_seq=0 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa239ffb640 1 -- 172.21.15.145:0/57381152 <== mds.3 v2:172.21.15.164:6836/1634173775 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563e564f7160 con 0x563e5659b400 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa239ffb640 10 client.25004 handle_command_reply: tid=0 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 client.25004 shutdown 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 msgr2=0x563e5658ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02c710 tx=0x7fa24c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 shutdown_connections 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e5659b400 0x563e5659a4e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e56596200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa239ffb640 0 client.25004 ms_handle_reset on v2:172.21.15.164:6836/1634173775 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56585130 0x563e5658f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 --2- 172.21.15.145:0/57381152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e5658ec80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 >> 172.21.15.145:0/57381152 conn(0x563e5654ca60 msgr2=0x563e5657f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 shutdown_connections 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 1 -- 172.21.15.145:0/57381152 wait complete. 2024-03-28T18:37:39.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:38.996+0000 7fa25ae79740 20 client.25004 trim_cache size 0 max 16384 2024-03-28T18:37:39.017 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 Processor -- start 2024-03-28T18:37:39.017 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 -- start start 2024-03-28T18:37:39.017 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56584740 0x563e5658d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.017 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56585130 0x563e5658dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659ac20 con 0x563e5658a2a0 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659b280 con 0x563e56585130 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa25ae79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659b8e0 con 0x563e56584740 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59898/0 (socket says 172.21.15.145:59898) 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 -- 172.21.15.145:0/2767739116 learned_addr learned my addr 172.21.15.145:0/2767739116 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256021640 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56584740 0x563e5658d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa255820640 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56585130 0x563e5658dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 -- 172.21.15.145:0/2767739116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56584740 msgr2=0x563e5658d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56584740 0x563e5658d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 -- 172.21.15.145:0/2767739116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56585130 msgr2=0x563e5658dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56585130 0x563e5658dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.018 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 -- 172.21.15.145:0/2767739116 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e562c86b0 con 0x563e5658a2a0 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa256822640 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2480ccfe0 tx=0x7fa24809bbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99f46dc29501cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.013+0000 7fa23bfff640 1 -- 172.21.15.145:0/2767739116 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b9070 con 0x563e5658a2a0 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa23bfff640 1 -- 172.21.15.145:0/2767739116 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa248099e10 con 0x563e5658a2a0 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa23bfff640 1 -- 172.21.15.145:0/2767739116 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b4070 con 0x563e5658a2a0 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- 172.21.15.145:0/2767739116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 msgr2=0x563e565961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2480ccfe0 tx=0x7fa24809bbd0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- 172.21.15.145:0/2767739116 shutdown_connections 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e5658a2a0 0x563e565961c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56585130 0x563e5658dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 --2- 172.21.15.145:0/2767739116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e56584740 0x563e5658d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- 172.21.15.145:0/2767739116 >> 172.21.15.145:0/2767739116 conn(0x563e5654ca60 msgr2=0x563e565866d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- 172.21.15.145:0/2767739116 shutdown_connections 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- 172.21.15.145:0/2767739116 wait complete. 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 -- start start 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.014+0000 7fa25ae79740 1 Processor -- start 2024-03-28T18:37:39.019 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56595450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e56596010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659c5a0 con 0x563e56585130 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659bf40 con 0x563e56584740 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e5659cc00 con 0x563e5658a2a0 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 10 client.0 _ll_register_callbacks cb 0x7fa24c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59914/0 (socket says 172.21.15.145:59914) 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 -- 172.21.15.145:0/1540627975 learned_addr learned my addr 172.21.15.145:0/1540627975 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa256021640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56595450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa256822640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e56596010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 -- 172.21.15.145:0/1540627975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 msgr2=0x563e56596010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e56596010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 -- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 msgr2=0x563e56595450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.020 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56595450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 -- 172.21.15.145:0/1540627975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e565f64b0 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa255820640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa240009730 tx=0x7fa240005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db6bdd3b9f6cbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa239ffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240005200 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa248099e10 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa2400053a0 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24003d050 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 10 client.15249 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.015+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563e565f77b0 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa248099e10 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa24001d020 con 0x563e56585130 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 10 client.15249 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 10 client.15249 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 20 client.15249 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e565fbba0 0x563e565f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.021 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 4 client.15249 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 --> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563e564f14b0 con 0x563e565fbba0 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa256021640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e565fbba0 0x563e565f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa239ffb640 10 client.15249 ms_handle_connect on v2:172.21.15.164:6836/1634173775 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa256021640 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e565fbba0 0x563e565f7a40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a87725c3aac6af01 server_cookie=75ecd2bec64e696b in_seq=0 out_seq=0 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa239ffb640 1 -- 172.21.15.145:0/1540627975 <== mds.3 v2:172.21.15.164:6836/1634173775 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x563e564f14b0 con 0x563e565fbba0 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa239ffb640 10 client.15249 handle_command_reply: tid=0 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.016+0000 7fa25ae79740 1 client.15249 shutdown 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 msgr2=0x563e56595990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa240009730 tx=0x7fa240005fd0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 shutdown_connections 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x563e565fbba0 0x563e565f7a40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563e5658a2a0 0x563e56596010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563e56585130 0x563e56595990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 --2- 172.21.15.145:0/1540627975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e56584740 0x563e56595450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 >> 172.21.15.145:0/1540627975 conn(0x563e5654ca60 msgr2=0x563e5657fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa239ffb640 0 client.15249 ms_handle_reset on v2:172.21.15.164:6836/1634173775 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 shutdown_connections 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 1 -- 172.21.15.145:0/1540627975 wait complete. 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa25ae79740 20 client.15249 trim_cache size 0 max 16384 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 3, 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24461, 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 87.93000558, 2024-03-28T18:37:39.023 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:37:39.024 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 93, 2024-03-28T18:37:39.024 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:37:39.024 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 126.274940994 2024-03-28T18:37:39.024 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:37:39.024 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 msgr2=0x7fa23c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa24c02c8e0 tx=0x7fa24c02e040 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 msgr2=0x7fa25019ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa248009fd0 tx=0x7fa24800ef10 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 shutdown_connections 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa250102150 0x7fa25019ed70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa250101790 0x7fa25019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa23c0a2800 0x7fa23c0a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 --2- 172.21.15.145:0/129355539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2501005a0 0x7fa250069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 >> 172.21.15.145:0/129355539 conn(0x7fa2500fbb30 msgr2=0x7fa250104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 shutdown_connections 2024-03-28T18:37:39.025 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.017+0000 7fa258285640 1 -- 172.21.15.145:0/129355539 wait complete. 2024-03-28T18:37:39.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 -- 172.21.15.145:0/12406062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 msgr2=0x7f3134101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 --2- 172.21.15.145:0/12406062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134101c10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f311c009900 tx=0x7f311c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 -- 172.21.15.145:0/12406062 shutdown_connections 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 --2- 172.21.15.145:0/12406062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 0x7f31341089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 --2- 172.21.15.145:0/12406062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134101c10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 --2- 172.21.15.145:0/12406062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 0x7f31341009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 -- 172.21.15.145:0/12406062 >> 172.21.15.145:0/12406062 conn(0x7f31340fbb30 msgr2=0x7f31340fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 -- 172.21.15.145:0/12406062 shutdown_connections 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.136+0000 7f31398e8640 1 -- 172.21.15.145:0/12406062 wait complete. 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 Processor -- start 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 -- start start 2024-03-28T18:37:39.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 0x7f31341979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 0x7f313419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313410c230 con 0x7f3134101790 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313410c0b0 con 0x7f3134102150 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31398e8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313410c3b0 con 0x7f31341005a0 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59928/0 (socket says 172.21.15.145:59928) 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 -- 172.21.15.145:0/4120222984 learned_addr learned my addr 172.21.15.145:0/4120222984 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 0x7f31341979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f31337fe640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 0x7f313419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 -- 172.21.15.145:0/4120222984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 msgr2=0x7f31341979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 0x7f31341979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 -- 172.21.15.145:0/4120222984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 msgr2=0x7f313419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 0x7f313419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.137+0000 7f312bfff640 1 -- 172.21.15.145:0/4120222984 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313419f600 con 0x7f3134101790 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f312bfff640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f311c009230 tx=0x7f311c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f31388e6640 1 -- 172.21.15.145:0/4120222984 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c041070 con 0x7f3134101790 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f31388e6640 1 -- 172.21.15.145:0/4120222984 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f311c004450 con 0x7f3134101790 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313419f890 con 0x7f3134101790 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f31388e6640 1 -- 172.21.15.145:0/4120222984 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c003e20 con 0x7f3134101790 2024-03-28T18:37:39.140 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.138+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313419fd80 con 0x7f3134101790 2024-03-28T18:37:39.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.140+0000 7f31388e6640 1 -- 172.21.15.145:0/4120222984 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f311c044070 con 0x7f3134101790 2024-03-28T18:37:39.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.140+0000 7f31388e6640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.140+0000 7f313c5aa740 1 Processor -- start 2024-03-28T18:37:39.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 -- start start 2024-03-28T18:37:39.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31388e6640 1 -- 172.21.15.145:0/4120222984 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f311c0f0950 con 0x7f3134101790 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc74cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc754730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc6b1c20 con 0x5598dc74c740 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc65aa40 con 0x5598dc7522a0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f313c5aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc6adcc0 con 0x5598dc74d130 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f312bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc754730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f312bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52416/0 (socket says 172.21.15.145:52416) 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f312bfff640 1 -- 172.21.15.145:0/687099994 learned_addr learned my addr 172.21.15.145:0/687099994 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f3132ffd640 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc74cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 0x7f31200a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f31341015f0 tx=0x7f3124011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 -- 172.21.15.145:0/687099994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 msgr2=0x5598dc751c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 -- 172.21.15.145:0/687099994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 msgr2=0x5598dc74cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc74cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 -- 172.21.15.145:0/687099994 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598dc67cf90 con 0x5598dc7522a0 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f312bfff640 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f31337fe640 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc754730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f312c009f00 tx=0x7f312c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a4e861b096bf23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.141+0000 7f3129ffb640 1 -- 172.21.15.145:0/687099994 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c004d20 con 0x5598dc7522a0 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f3129ffb640 1 -- 172.21.15.145:0/687099994 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312c004030 con 0x5598dc7522a0 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f3129ffb640 1 -- 172.21.15.145:0/687099994 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c005260 con 0x5598dc7522a0 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- 172.21.15.145:0/687099994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 msgr2=0x5598dc754730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc754730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f312c009f00 tx=0x7f312c011810 comp rx=0 tx=0).stop 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- 172.21.15.145:0/687099994 shutdown_connections 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc754730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc751c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.148 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 --2- 172.21.15.145:0/687099994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc74cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- 172.21.15.145:0/687099994 >> 172.21.15.145:0/687099994 conn(0x5598dc714a60 msgr2=0x5598dc7480b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- 172.21.15.145:0/687099994 shutdown_connections 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- 172.21.15.145:0/687099994 wait complete. 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.142+0000 7f313c5aa740 1 -- start start 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 Processor -- start 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 0x5598dc757190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 0x5598dc75e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc755250 con 0x5598dc74c740 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc66de90 con 0x5598dc74d130 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc7553d0 con 0x5598dc7522a0 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f313c5aa740 10 client.0 _ll_register_callbacks cb 0x5598dc3c5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59936/0 (socket says 172.21.15.145:59936) 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f31337fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 0x5598dc75e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 -- 172.21.15.145:0/3796967752 learned_addr learned my addr 172.21.15.145:0/3796967752 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f312bfff640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 0x5598dc757190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 -- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 msgr2=0x5598dc75e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 0x5598dc75e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 -- 172.21.15.145:0/3796967752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 msgr2=0x5598dc757190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 0x5598dc757190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f3132ffd640 1 -- 172.21.15.145:0/3796967752 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598dc67a9d0 con 0x5598dc74c740 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f31337fe640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 0x5598dc75e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.143+0000 7f312bfff640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 0x5598dc757190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3132ffd640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f312400fa50 tx=0x7f3124004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563a77dad83d155a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124017040 con 0x5598dc74c740 2024-03-28T18:37:39.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3124004020 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f312c004030 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3124004220 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598dc3c62d0 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312c004030 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3124025070 con 0x5598dc74c740 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 10 client.15261 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 20 client.15261 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7634e0 0x5598dc762510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 4 client.15261 mds_command: new command op to 24469 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.144+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 --> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598dc6bf160 con 0x5598dc7634e0 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f312bfff640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7634e0 0x5598dc762510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f312bfff640 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7634e0 0x5598dc762510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=63164f848f2577af server_cookie=b196da370747fc40 in_seq=0 out_seq=0 2024-03-28T18:37:39.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f3112ffd640 10 client.15261 ms_handle_connect on v2:172.21.15.192:6836/2153191464 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f3112ffd640 1 -- 172.21.15.145:0/3796967752 <== mds.2 v2:172.21.15.192:6836/2153191464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598dc6bf160 con 0x5598dc7634e0 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f3112ffd640 10 client.15261 handle_command_reply: tid=0 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.145+0000 7f313c5aa740 1 client.15261 shutdown 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 msgr2=0x5598dc756c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f312400fa50 tx=0x7f3124004730 comp rx=0 tx=0).stop 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 shutdown_connections 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc7522a0 0x5598dc75e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74d130 0x5598dc757190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7634e0 0x5598dc762510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 --2- 172.21.15.145:0/3796967752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc756c50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f3112ffd640 0 client.15261 ms_handle_reset on v2:172.21.15.192:6836/2153191464 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 >> 172.21.15.145:0/3796967752 conn(0x5598dc714a60 msgr2=0x5598dc7472f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 shutdown_connections 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 1 -- 172.21.15.145:0/3796967752 wait complete. 2024-03-28T18:37:39.152 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.146+0000 7f313c5aa740 20 client.15261 trim_cache size 0 max 16384 2024-03-28T18:37:39.164 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 Processor -- start 2024-03-28T18:37:39.164 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 -- start start 2024-03-28T18:37:39.164 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc756860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc762d00 con 0x5598dc74c740 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc763360 con 0x5598dc7522a0 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f313c5aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc7639c0 con 0x5598dc74d130 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59942/0 (socket says 172.21.15.145:59942) 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 -- 172.21.15.145:0/1174345318 learned_addr learned my addr 172.21.15.145:0/1174345318 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f312bfff640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f31337fe640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc756860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 -- 172.21.15.145:0/1174345318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 msgr2=0x5598dc75e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 -- 172.21.15.145:0/1174345318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 msgr2=0x5598dc756860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc756860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 -- 172.21.15.145:0/1174345318 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598dc751ff0 con 0x5598dc74c740 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f312bfff640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.163+0000 7f3132ffd640 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3124009a60 tx=0x7f3124005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea25ab70f5a9108 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f31297fa640 1 -- 172.21.15.145:0/1174345318 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124017070 con 0x5598dc74c740 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f31297fa640 1 -- 172.21.15.145:0/1174345318 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3124019050 con 0x5598dc74c740 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f31297fa640 1 -- 172.21.15.145:0/1174345318 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240338d0 con 0x5598dc74c740 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 -- 172.21.15.145:0/1174345318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 msgr2=0x5598dc762510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3124009a60 tx=0x7f3124005d30 comp rx=0 tx=0).stop 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 -- 172.21.15.145:0/1174345318 shutdown_connections 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc7522a0 0x5598dc756860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 --2- 172.21.15.145:0/1174345318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc74c740 0x5598dc762510 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 -- 172.21.15.145:0/1174345318 >> 172.21.15.145:0/1174345318 conn(0x5598dc714a60 msgr2=0x5598dc74db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 -- 172.21.15.145:0/1174345318 shutdown_connections 2024-03-28T18:37:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.164+0000 7f313c5aa740 1 -- 172.21.15.145:0/1174345318 wait complete. 2024-03-28T18:37:39.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 -- start start 2024-03-28T18:37:39.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 Processor -- start 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 0x5598dc759c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc764680 con 0x5598dc7522a0 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc764020 con 0x5598dc74c740 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598dc764ce0 con 0x5598dc74d130 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f313c5aa740 10 client.0 _ll_register_callbacks cb 0x7f312400f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f31337fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f31337fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59954/0 (socket says 172.21.15.145:59954) 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f31337fe640 1 -- 172.21.15.145:0/4003785941 learned_addr learned my addr 172.21.15.145:0/4003785941 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 0x5598dc759c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.165+0000 7f312bfff640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 -- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 msgr2=0x5598dc75a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 -- 172.21.15.145:0/4003785941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 msgr2=0x5598dc75d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 -- 172.21.15.145:0/4003785941 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598dc4906b0 con 0x5598dc74c740 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f31337fe640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3132ffd640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 0x5598dc759c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3124006010 tx=0x7f312401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9d92e693ba3025 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124017040 con 0x5598dc74c740 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3124004680 con 0x5598dc74c740 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3124019050 con 0x5598dc74c740 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3124047050 con 0x5598dc74c740 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 10 client.25028 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598dc7be750 con 0x5598dc74c740 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3124019050 con 0x5598dc74c740 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3124025040 con 0x5598dc74c740 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 10 client.25028 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 10 client.25028 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 20 client.25028 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7c2b00 0x5598dc7be920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 4 client.25028 mds_command: new command op to 24469 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.166+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 --> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5598dc6b94b0 con 0x5598dc7c2b00 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f312bfff640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7c2b00 0x5598dc7be920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f312bfff640 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7c2b00 0x5598dc7be920 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=600347436e6b8d18 server_cookie=49db9929b8dbd4da in_seq=0 out_seq=0 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f3112ffd640 10 client.25028 ms_handle_connect on v2:172.21.15.192:6836/2153191464 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f3112ffd640 1 -- 172.21.15.145:0/4003785941 <== mds.2 v2:172.21.15.192:6836/2153191464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5598dc6b94b0 con 0x5598dc7c2b00 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f3112ffd640 10 client.25028 handle_command_reply: tid=0 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 client.25028 shutdown 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 msgr2=0x5598dc759c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 0x5598dc759c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3124006010 tx=0x7f312401c000 comp rx=0 tx=0).stop 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 shutdown_connections 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x5598dc7c2b00 0x5598dc7be920 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5598dc7522a0 0x5598dc75d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f3112ffd640 0 client.25028 ms_handle_reset on v2:172.21.15.192:6836/2153191464 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5598dc74d130 0x5598dc75a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 --2- 172.21.15.145:0/4003785941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5598dc74c740 0x5598dc759c60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 >> 172.21.15.145:0/4003785941 conn(0x5598dc714a60 msgr2=0x5598dc762510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 shutdown_connections 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.167+0000 7f313c5aa740 1 -- 172.21.15.145:0/4003785941 wait complete. 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f313c5aa740 20 client.25028 trim_cache size 0 max 16384 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:37:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 2, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24469, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 89.084432863999993, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 93, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 129.40675434400001 2024-03-28T18:37:39.174 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:37:39.177 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 msgr2=0x7f31200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.177 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 0x7f31200a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f31341015f0 tx=0x7f3124011440 comp rx=0 tx=0).stop 2024-03-28T18:37:39.177 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 msgr2=0x7f3134197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f311c009230 tx=0x7f311c02c650 comp rx=0 tx=0).stop 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 shutdown_connections 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3134102150 0x7f313419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3134101790 0x7f3134197f20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f31200a2800 0x7f31200a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 --2- 172.21.15.145:0/4120222984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31341005a0 0x7f31341979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 >> 172.21.15.145:0/4120222984 conn(0x7f31340fbb30 msgr2=0x7f31340fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 shutdown_connections 2024-03-28T18:37:39.178 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.168+0000 7f31398e8640 1 -- 172.21.15.145:0/4120222984 wait complete. 2024-03-28T18:37:39.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:37:39.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.283+0000 7f7b471f4640 1 -- 172.21.15.145:0/1209606295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40104a50 msgr2=0x7f7b40106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.283+0000 7f7b471f4640 1 --2- 172.21.15.145:0/1209606295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40104a50 0x7f7b40106e40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009900 tx=0x7f7b2c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- 172.21.15.145:0/1209606295 shutdown_connections 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- 172.21.15.145:0/1209606295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40107380 0x7f7b401097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- 172.21.15.145:0/1209606295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40104a50 0x7f7b40106e40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- 172.21.15.145:0/1209606295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40102120 0x7f7b40104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- 172.21.15.145:0/1209606295 >> 172.21.15.145:0/1209606295 conn(0x7f7b400fbb30 msgr2=0x7f7b400fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- 172.21.15.145:0/1209606295 shutdown_connections 2024-03-28T18:37:39.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- 172.21.15.145:0/1209606295 wait complete. 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 Processor -- start 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- start start 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 0x7f7b40197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40104a50 0x7f7b40197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4010d360 con 0x7f7b40107380 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4010d1e0 con 0x7f7b40102120 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b471f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4010d4e0 con 0x7f7b40104a50 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b45791640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 0x7f7b40197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b45791640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59970/0 (socket says 172.21.15.145:59970) 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 0x7f7b40197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51712/0 (socket says 172.21.15.145:51712) 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b45791640 1 -- 172.21.15.145:0/992497183 learned_addr learned my addr 172.21.15.145:0/992497183 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.284+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40104a50 0x7f7b40197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 -- 172.21.15.145:0/992497183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40104a50 msgr2=0x7f7b40197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40104a50 0x7f7b40197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 -- 172.21.15.145:0/992497183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 msgr2=0x7f7b40197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 0x7f7b40197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 -- 172.21.15.145:0/992497183 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4019f630 con 0x7f7b40107380 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b45791640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800efd0 tx=0x7f7b3800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b3dffb640 1 -- 172.21.15.145:0/992497183 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38003b70 con 0x7f7b40107380 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b3dffb640 1 -- 172.21.15.145:0/992497183 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b38003d10 con 0x7f7b40107380 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4019f920 con 0x7f7b40107380 2024-03-28T18:37:39.287 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b3dffb640 1 -- 172.21.15.145:0/992497183 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b380109f0 con 0x7f7b40107380 2024-03-28T18:37:39.288 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.285+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4019fe40 con 0x7f7b40107380 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.287+0000 7f7b3dffb640 1 -- 172.21.15.145:0/992497183 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7b38004020 con 0x7f7b40107380 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.288+0000 7f7b3dffb640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 0x7f7b300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.288+0000 7f7b44f90640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 0x7f7b300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.288+0000 7f7b3dffb640 1 -- 172.21.15.145:0/992497183 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7b380cf1b0 con 0x7f7b40107380 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.288+0000 7f7b44f90640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 0x7f7b300a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009bd0 tx=0x7f7b34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 Processor -- start 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 -- start start 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 0x561e1b2bf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b21cc20 con 0x561e1b2b8130 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b1c5a40 con 0x561e1b2bd2a0 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b49de5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b218cc0 con 0x561e1b2b7740 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52456/0 (socket says 172.21.15.145:52456) 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59976/0 (socket says 172.21.15.145:59976) 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 -- 172.21.15.145:0/2193129802 learned_addr learned my addr 172.21.15.145:0/2193129802 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b45791640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 0x561e1b2bf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 -- 172.21.15.145:0/2193129802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 msgr2=0x561e1b2bf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 0x561e1b2bf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 -- 172.21.15.145:0/2193129802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 msgr2=0x561e1b2bcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 -- 172.21.15.145:0/2193129802 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e1b1e7f90 con 0x561e1b2b7740 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b45791640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 0x561e1b2bf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.289+0000 7f7b44f90640 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e630 tx=0x7f7b34014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3ce397741082ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.290+0000 7f7b2affd640 1 -- 172.21.15.145:0/2193129802 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34004020 con 0x561e1b2b7740 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.290+0000 7f7b2affd640 1 -- 172.21.15.145:0/2193129802 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b340041c0 con 0x561e1b2b7740 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.290+0000 7f7b2affd640 1 -- 172.21.15.145:0/2193129802 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3401c720 con 0x561e1b2b7740 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.290+0000 7f7b49de5740 1 -- 172.21.15.145:0/2193129802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 msgr2=0x561e1b2b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.290+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e630 tx=0x7f7b34014000 comp rx=0 tx=0).stop 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 -- 172.21.15.145:0/2193129802 shutdown_connections 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2bd2a0 0x561e1b2bf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b8130 0x561e1b2bcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2193129802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b7740 0x561e1b2b7b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 -- 172.21.15.145:0/2193129802 >> 172.21.15.145:0/2193129802 conn(0x561e1b27fa60 msgr2=0x561e1b2b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 -- 172.21.15.145:0/2193129802 shutdown_connections 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 -- 172.21.15.145:0/2193129802 wait complete. 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 -- start start 2024-03-28T18:37:39.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 Processor -- start 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b7740 0x561e1b2c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 0x561e1b2c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.291+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2c0250 con 0x561e1b2bd2a0 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b1d8e90 con 0x561e1b2b7740 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2c03d0 con 0x561e1b2b8130 2024-03-28T18:37:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 10 client.0 _ll_register_callbacks cb 0x561e1af30ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 0x561e1b2c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b7740 0x561e1b2c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59990/0 (socket says 172.21.15.145:59990) 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 -- 172.21.15.145:0/2982690420 learned_addr learned my addr 172.21.15.145:0/2982690420 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 -- 172.21.15.145:0/2982690420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 msgr2=0x561e1b2c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 0x561e1b2c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 -- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b7740 msgr2=0x561e1b2c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b7740 0x561e1b2c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 -- 172.21.15.145:0/2982690420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e1b1e59d0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 0x561e1b2c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b45791640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b380981c0 tx=0x7f7b3809b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ccd19444770253 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3809b7f0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.298 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7b38004620 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b340041c0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b3809f940 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.292+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561e1af312d0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b340041c0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7b380bb040 con 0x561e1b2bd2a0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 10 client.15279 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 20 client.15279 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b2ce3d0 0x561e1b2cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 4 client.15279 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 --> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e1b22a160 con 0x561e1b2ce3d0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b44f90640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b2ce3d0 0x561e1b2cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b1ffff640 10 client.15279 ms_handle_connect on v2:172.21.15.164:6838/644460294 2024-03-28T18:37:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.293+0000 7f7b44f90640 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b2ce3d0 0x561e1b2cd520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5ca781c2800da59 server_cookie=59645575aebd7867 in_seq=0 out_seq=0 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.294+0000 7f7b1ffff640 1 -- 172.21.15.145:0/2982690420 <== mds.1 v2:172.21.15.164:6838/644460294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e1b22a160 con 0x561e1b2ce3d0 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.294+0000 7f7b1ffff640 10 client.15279 handle_command_reply: tid=0 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.294+0000 7f7b49de5740 1 client.15279 shutdown 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 msgr2=0x561e1b2c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7b380981c0 tx=0x7f7b3809b210 comp rx=0 tx=0).stop 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 shutdown_connections 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b2ce3d0 0x561e1b2cd520 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c91d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b1ffff640 0 client.15279 ms_handle_reset on v2:172.21.15.164:6838/644460294 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2b8130 0x561e1b2c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 --2- 172.21.15.145:0/2982690420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b7740 0x561e1b2c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 >> 172.21.15.145:0/2982690420 conn(0x561e1b27fa60 msgr2=0x561e1b2b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 shutdown_connections 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 1 -- 172.21.15.145:0/2982690420 wait complete. 2024-03-28T18:37:39.300 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.295+0000 7f7b49de5740 20 client.15279 trim_cache size 0 max 16384 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.311+0000 7f7b49de5740 1 Processor -- start 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.311+0000 7f7b49de5740 1 -- start start 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2cdbf0 con 0x561e1b2b7740 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2ce250 con 0x561e1b2b8130 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b49de5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2ce8b0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60004/0 (socket says 172.21.15.145:60004) 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 -- 172.21.15.145:0/646238691 learned_addr learned my addr 172.21.15.145:0/646238691 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b45791640 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 -- 172.21.15.145:0/646238691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 msgr2=0x561e1b2b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 -- 172.21.15.145:0/646238691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 msgr2=0x561e1b2c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 -- 172.21.15.145:0/646238691 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e1affb6b0 con 0x561e1b2b7740 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b44f90640 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e630 tx=0x7f7b34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aefb6daf3df56c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b2a7fc640 1 -- 172.21.15.145:0/646238691 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34034280 con 0x561e1b2b7740 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b2a7fc640 1 -- 172.21.15.145:0/646238691 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3401c410 con 0x561e1b2b7740 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.312+0000 7f7b2a7fc640 1 -- 172.21.15.145:0/646238691 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3401cd10 con 0x561e1b2b7740 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- 172.21.15.145:0/646238691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 msgr2=0x561e1b2b62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400e630 tx=0x7f7b34017000 comp rx=0 tx=0).stop 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- 172.21.15.145:0/646238691 shutdown_connections 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 --2- 172.21.15.145:0/646238691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2b62f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- 172.21.15.145:0/646238691 >> 172.21.15.145:0/646238691 conn(0x561e1b27fa60 msgr2=0x561e1b2b9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- 172.21.15.145:0/646238691 shutdown_connections 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- 172.21.15.145:0/646238691 wait complete. 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 -- start start 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.313+0000 7f7b49de5740 1 Processor -- start 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2cf570 con 0x561e1b2b7740 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2cef10 con 0x561e1b2b8130 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e1b2cfbd0 con 0x561e1b2bd2a0 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 10 client.0 _ll_register_callbacks cb 0x7f7b38097450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60006/0 (socket says 172.21.15.145:60006) 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 -- 172.21.15.145:0/260742968 learned_addr learned my addr 172.21.15.145:0/260742968 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b45791640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 -- 172.21.15.145:0/260742968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 msgr2=0x561e1b2c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 -- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 msgr2=0x561e1b2c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 -- 172.21.15.145:0/260742968 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e1b2c8e60 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b44f90640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400a010 tx=0x7f7b340096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247c0f1587950570 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3401c7c0 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7b340340c0 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b3401c410 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b34047050 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 10 client.15285 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.314+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561e1b329e70 con 0x561e1b2b7740 2024-03-28T18:37:39.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b3401c410 con 0x561e1b2b7740 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7b34034770 con 0x561e1b2b7740 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 10 client.15285 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 10 client.15285 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 20 client.15285 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b32dca0 0x561e1b32a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 4 client.15285 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 --> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561e1b2244b0 con 0x561e1b32dca0 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b32dca0 0x561e1b32a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b1ffff640 10 client.15285 ms_handle_connect on v2:172.21.15.164:6838/644460294 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.315+0000 7f7b3ffff640 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b32dca0 0x561e1b32a040 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f495012772ad58ef server_cookie=8bcc98f69b6cb5b2 in_seq=0 out_seq=0 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b1ffff640 1 -- 172.21.15.145:0/260742968 <== mds.1 v2:172.21.15.164:6838/644460294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561e1b2244b0 con 0x561e1b32dca0 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b1ffff640 10 client.15285 handle_command_reply: tid=0 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 client.15285 shutdown 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 msgr2=0x561e1b2c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b3400a010 tx=0x7f7b340096c0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.320 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 shutdown_connections 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x561e1b32dca0 0x561e1b32a040 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e1b2bd2a0 0x561e1b2c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e1b2b8130 0x561e1b2c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 --2- 172.21.15.145:0/260742968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e1b2b7740 0x561e1b2c5310 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b1ffff640 0 client.15285 ms_handle_reset on v2:172.21.15.164:6838/644460294 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 >> 172.21.15.145:0/260742968 conn(0x561e1b27fa60 msgr2=0x561e1b2b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 shutdown_connections 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 1 -- 172.21.15.145:0/260742968 wait complete. 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.316+0000 7f7b49de5740 20 client.15285 trim_cache size 0 max 16384 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:37:39.321 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 1, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24491, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 90.242924926000001, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 93, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 115.066473348 2024-03-28T18:37:39.322 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 msgr2=0x7f7b300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 0x7f7b300a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009bd0 tx=0x7f7b34005f90 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 msgr2=0x7f7b4019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800efd0 tx=0x7f7b3800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 shutdown_connections 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b40107380 0x7f7b4019efc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7b300a2880 0x7f7b300a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b40104a50 0x7f7b40197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 --2- 172.21.15.145:0/992497183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b40102120 0x7f7b40197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 >> 172.21.15.145:0/992497183 conn(0x7f7b400fbb30 msgr2=0x7f7b40102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 shutdown_connections 2024-03-28T18:37:39.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.317+0000 7f7b471f4640 1 -- 172.21.15.145:0/992497183 wait complete. 2024-03-28T18:37:39.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 -- 172.21.15.145:0/879384647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 msgr2=0x7f96c81003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 --2- 172.21.15.145:0/879384647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c81003b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009ca0 tx=0x7f96c002c180 comp rx=0 tx=0).stop 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 -- 172.21.15.145:0/879384647 shutdown_connections 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 --2- 172.21.15.145:0/879384647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 0x7f96c8101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 --2- 172.21.15.145:0/879384647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c8100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 --2- 172.21.15.145:0/879384647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c81003b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 -- 172.21.15.145:0/879384647 >> 172.21.15.145:0/879384647 conn(0x7f96c80fbb30 msgr2=0x7f96c80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 -- 172.21.15.145:0/879384647 shutdown_connections 2024-03-28T18:37:39.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.431+0000 7f96cce43640 1 -- 172.21.15.145:0/879384647 wait complete. 2024-03-28T18:37:39.433 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 Processor -- start 2024-03-28T18:37:39.433 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 -- start start 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 0x7f96c81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c810f510 con 0x7f96c81008f0 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c810f390 con 0x7f96c80fff90 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96cce43640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c810f690 con 0x7f96c8101290 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60016/0 (socket says 172.21.15.145:60016) 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c5d9b640 1 -- 172.21.15.145:0/3761905915 learned_addr learned my addr 172.21.15.145:0/3761905915 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 0x7f96c81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 -- 172.21.15.145:0/3761905915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 msgr2=0x7f96c81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 0x7f96c81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 -- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 msgr2=0x7f96c819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c659c640 1 -- 172.21.15.145:0/3761905915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c81a1810 con 0x7f96c80fff90 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.432+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96c659c640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c8199bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96c002c6c0 tx=0x7f96c0009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96b77fe640 1 -- 172.21.15.145:0/3761905915 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0039040 con 0x7f96c80fff90 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c81a1aa0 con 0x7f96c80fff90 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96b77fe640 1 -- 172.21.15.145:0/3761905915 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c0003e50 con 0x7f96c80fff90 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96b77fe640 1 -- 172.21.15.145:0/3761905915 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0004280 con 0x7f96c80fff90 2024-03-28T18:37:39.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.433+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c81a1fe0 con 0x7f96c80fff90 2024-03-28T18:37:39.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.435+0000 7f96b77fe640 1 -- 172.21.15.145:0/3761905915 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96c0046030 con 0x7f96c80fff90 2024-03-28T18:37:39.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.435+0000 7f96cfa2a740 1 Processor -- start 2024-03-28T18:37:39.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 -- start start 2024-03-28T18:37:39.441 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748dc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555974839c20 con 0x5559748da2a0 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559747e2a40 con 0x5559748d4740 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555974835cc0 con 0x5559748d5130 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51786/0 (socket says 172.21.15.145:51786) 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c6d9d640 1 -- 172.21.15.145:0/14009734 learned_addr learned my addr 172.21.15.145:0/14009734 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96b77fe640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 0x7f96b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 -- 172.21.15.145:0/14009734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 msgr2=0x5559748d9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748d9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 -- 172.21.15.145:0/14009734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 msgr2=0x5559748d4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 -- 172.21.15.145:0/14009734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555974804f90 con 0x5559748da2a0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 0x7f96b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96b77fe640 1 -- 172.21.15.145:0/3761905915 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f96c00f0870 con 0x7f96c80fff90 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.436+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748dc730 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0114c0 tx=0x7f96bc011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898920df6d7837ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96b4ff9640 1 -- 172.21.15.145:0/14009734 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc004030 con 0x5559748da2a0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96b4ff9640 1 -- 172.21.15.145:0/14009734 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96bc0041d0 con 0x5559748da2a0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96b4ff9640 1 -- 172.21.15.145:0/14009734 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00f9c0 con 0x5559748da2a0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 -- 172.21.15.145:0/14009734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 msgr2=0x5559748dc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748dc730 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0114c0 tx=0x7f96bc011980 comp rx=0 tx=0).stop 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 0x7f96b00a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5559748d5990 tx=0x7f96b80046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.443 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 -- 172.21.15.145:0/14009734 shutdown_connections 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748dc730 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748d9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/14009734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.437+0000 7f96cfa2a740 1 -- 172.21.15.145:0/14009734 >> 172.21.15.145:0/14009734 conn(0x55597489ca60 msgr2=0x5559748d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- 172.21.15.145:0/14009734 shutdown_connections 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- 172.21.15.145:0/14009734 wait complete. 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- start start 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 Processor -- start 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 0x5559748dec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748e61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748dd250 con 0x5559748da2a0 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559747f5e90 con 0x5559748d5130 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748dd3d0 con 0x5559748d4740 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96cfa2a740 10 client.0 _ll_register_callbacks cb 0x55597454dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.444 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 0x5559748dec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c6d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748e61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:51796/0 (socket says 172.21.15.145:51796) 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 0x5559748dec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52504/0 (socket says 172.21.15.145:52504) 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c5d9b640 1 -- 172.21.15.145:0/3758241391 learned_addr learned my addr 172.21.15.145:0/3758241391 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c6d9d640 1 -- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 msgr2=0x5559748dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 0x5559748dec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c6d9d640 1 -- 172.21.15.145:0/3758241391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 msgr2=0x5559748df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.438+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96c6d9d640 1 -- 172.21.15.145:0/3758241391 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559748029d0 con 0x5559748da2a0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748e61b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f96b800fc40 tx=0x7f96b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4b61ac3ec9074f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.445 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b802b060 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f96b80263f0 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96bc0041d0 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96b8032050 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55597454e2d0 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96bc0041d0 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f96b8025030 con 0x5559748da2a0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 10 client.15297 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 20 client.15297 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5559748eb460 0x5559748ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 4 client.15297 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.439+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555974847160 con 0x5559748eb460 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.440+0000 7f96c659c640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5559748eb460 0x5559748ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.446 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.440+0000 7f96adffb640 10 client.15297 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.440+0000 7f96c659c640 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5559748eb460 0x5559748ea4f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82e06d25acbc0f21 server_cookie=4a1a5390e4633e1 in_seq=0 out_seq=0 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96adffb640 1 -- 172.21.15.145:0/3758241391 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555974847160 con 0x5559748eb460 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96adffb640 10 client.15297 handle_command_reply: tid=0 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 client.15297 shutdown 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 msgr2=0x5559748e61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748e61b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f96b800fc40 tx=0x7f96b8004730 comp rx=0 tx=0).stop 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 shutdown_connections 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5559748eb460 0x5559748ea4f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748e61b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96adffb640 0 client.15297 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d5130 0x5559748df170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/3758241391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d4740 0x5559748dec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 >> 172.21.15.145:0/3758241391 conn(0x55597489ca60 msgr2=0x5559748cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 shutdown_connections 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 1 -- 172.21.15.145:0/3758241391 wait complete. 2024-03-28T18:37:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.441+0000 7f96cfa2a740 20 client.15297 trim_cache size 0 max 16384 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 Processor -- start 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 -- start start 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748e60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748eac80 con 0x5559748da2a0 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748eb2e0 con 0x5559748d4740 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748eb940 con 0x5559748d5130 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60048/0 (socket says 172.21.15.145:60048) 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 -- 172.21.15.145:0/2385518395 learned_addr learned my addr 172.21.15.145:0/2385518395 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.475 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748e60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 -- 172.21.15.145:0/2385518395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 msgr2=0x5559748e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748e60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 -- 172.21.15.145:0/2385518395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 msgr2=0x5559748d5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.470+0000 7f96c6d9d640 1 -- 172.21.15.145:0/2385518395 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559746186b0 con 0x5559748da2a0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009ad0 tx=0x7f96b8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7480c90395daf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96affff640 1 -- 172.21.15.145:0/2385518395 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b803b070 con 0x5559748da2a0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96affff640 1 -- 172.21.15.145:0/2385518395 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96b800fc70 con 0x5559748da2a0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96affff640 1 -- 172.21.15.145:0/2385518395 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b802b040 con 0x5559748da2a0 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2385518395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 msgr2=0x5559748d6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009ad0 tx=0x7f96b8011f90 comp rx=0 tx=0).stop 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2385518395 shutdown_connections 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748da2a0 0x5559748d6630 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748d5130 0x5559748e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2385518395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748d5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2385518395 >> 172.21.15.145:0/2385518395 conn(0x55597489ca60 msgr2=0x5559748ea4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2385518395 shutdown_connections 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.471+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2385518395 wait complete. 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 -- start start 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 Processor -- start 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748e2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748da2a0 0x5559748e5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748ec600 con 0x5559748d5130 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748ebfa0 con 0x5559748d4740 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559748ecc60 con 0x5559748da2a0 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96cfa2a740 10 client.0 _ll_register_callbacks cb 0x7f96b80099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60054/0 (socket says 172.21.15.145:60054) 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 -- 172.21.15.145:0/2380600114 learned_addr learned my addr 172.21.15.145:0/2380600114 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:39.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c659c640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c6d9d640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748da2a0 0x5559748e5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 -- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748da2a0 msgr2=0x5559748e5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748da2a0 0x5559748e5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 -- 172.21.15.145:0/2380600114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 msgr2=0x5559748e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c5d9b640 1 -- 172.21.15.145:0/2380600114 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559748e5b50 con 0x5559748d5130 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.472+0000 7f96c659c640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96c5d9b640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0097f0 tx=0x7f96bc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ba2c294e6d798d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc003be0 con 0x5559748d5130 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f96bc016040 con 0x5559748d5130 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96b800fc70 con 0x5559748d5130 2024-03-28T18:37:39.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96bc033050 con 0x5559748d5130 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555974946bb0 con 0x5559748d5130 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96b800fc70 con 0x5559748d5130 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f96bc0194e0 con 0x5559748d5130 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 20 client.15303 populate_metadata read hostname 'smithi145' 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55597494b090 0x555974946e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 4 client.15303 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5559748414b0 con 0x55597494b090 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.473+0000 7f96c659c640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55597494b090 0x555974946e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96c659c640 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55597494b090 0x555974946e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31998873e73583b5 server_cookie=f7a210fc838deea2 in_seq=0 out_seq=0 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96adffb640 10 client.15303 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96adffb640 1 -- 172.21.15.145:0/2380600114 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5559748414b0 con 0x55597494b090 2024-03-28T18:37:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96adffb640 10 client.15303 handle_command_reply: tid=0 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 client.15303 shutdown 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 msgr2=0x5559748e2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0097f0 tx=0x7f96bc003820 comp rx=0 tx=0).stop 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 shutdown_connections 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55597494b090 0x555974946e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559748da2a0 0x5559748e5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5559748d5130 0x5559748e2840 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 --2- 172.21.15.145:0/2380600114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5559748d4740 0x5559748e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96adffb640 0 client.15303 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 >> 172.21.15.145:0/2380600114 conn(0x55597489ca60 msgr2=0x5559748d5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 shutdown_connections 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 1 -- 172.21.15.145:0/2380600114 wait complete. 2024-03-28T18:37:39.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.474+0000 7f96cfa2a740 20 client.15303 trim_cache size 0 max 16384 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 0, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24496, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 108.556018154, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 93, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 123.959190623 2024-03-28T18:37:39.481 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:37:39.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 msgr2=0x7f96b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 0x7f96b00a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5559748d5990 tx=0x7f96b80046c0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 msgr2=0x7f96c8199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:39.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c8199bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96c002c6c0 tx=0x7f96c0009360 comp rx=0 tx=0).stop 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 shutdown_connections 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f96b00a2800 0x7f96b00a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f96c8101290 0x7f96c81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c81008f0 0x7f96c819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 --2- 172.21.15.145:0/3761905915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c80fff90 0x7f96c8199bf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 >> 172.21.15.145:0/3761905915 conn(0x7f96c80fbb30 msgr2=0x7f96c80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 shutdown_connections 2024-03-28T18:37:39.483 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:39.475+0000 7f96cce43640 1 -- 172.21.15.145:0/3761905915 wait complete. 2024-03-28T18:37:39.486 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T18:37:39.486 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:37:39.495 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:37:39.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T18:37:39.750 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:39.871 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:39 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3423987891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:39.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:39 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3423987891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:39.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:39 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3423987891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:37:40.903 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:40 smithi145 ceph-mon[32151]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:40.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:40 smithi192 ceph-mon[36589]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:40 smithi164 ceph-mon[36248]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 -- 172.21.15.145:0/2700103064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105610 msgr2=0x7fc238105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 --2- 172.21.15.145:0/2700103064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105610 0x7fc238105a70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc2280099c0 tx=0x7fc228009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 -- 172.21.15.145:0/2700103064 shutdown_connections 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 --2- 172.21.15.145:0/2700103064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238105fb0 0x7fc23810c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 --2- 172.21.15.145:0/2700103064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238104410 0x7fc238104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 --2- 172.21.15.145:0/2700103064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105610 0x7fc238105a70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 -- 172.21.15.145:0/2700103064 >> 172.21.15.145:0/2700103064 conn(0x7fc2380ff9b0 msgr2=0x7fc238101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:41.732 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 -- 172.21.15.145:0/2700103064 shutdown_connections 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.730+0000 7fc240130700 1 -- 172.21.15.145:0/2700103064 wait complete. 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 Processor -- start 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 -- start start 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 0x7fc23819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 0x7fc2381a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238110000 con 0x7fc238105fb0 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23810fe80 con 0x7fc238104410 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc240130700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238110180 con 0x7fc238105610 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:52550/0 (socket says 172.21.15.145:52550) 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23e6cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 0x7fc2381a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 -- 172.21.15.145:0/118201606 learned_addr learned my addr 172.21.15.145:0/118201606 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23decc700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 0x7fc23819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:41.733 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 -- 172.21.15.145:0/118201606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 msgr2=0x7fc23819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 0x7fc23819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 -- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 msgr2=0x7fc2381a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 0x7fc2381a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23d6cb700 1 -- 172.21.15.145:0/118201606 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2381a2210 con 0x7fc238105610 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23decc700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 0x7fc23819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.731+0000 7fc23e6cd700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 0x7fc2381a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc23d6cb700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc228000c00 tx=0x7fc22800bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc228003e10 con 0x7fc238105610 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc22800f460 con 0x7fc238105610 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2381a24a0 con 0x7fc238105610 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc228017650 con 0x7fc238105610 2024-03-28T18:37:41.734 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.732+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381a28f0 con 0x7fc238105610 2024-03-28T18:37:41.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.733+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc238068e50 con 0x7fc238105610 2024-03-28T18:37:41.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.734+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc22800f910 con 0x7fc238105610 2024-03-28T18:37:41.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.735+0000 7fc22effd700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 0x7fc2240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:41.735 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.735+0000 7fc23decc700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 0x7fc2240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:41.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.735+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc2280cfd50 con 0x7fc238105610 2024-03-28T18:37:41.736 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.736+0000 7fc23decc700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 0x7fc2240a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc238105470 tx=0x7fc234009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:41.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.739+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc228013070 con 0x7fc238105610 2024-03-28T18:37:41.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.961+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2380625d0 con 0x7fc2240a2890 2024-03-28T18:37:41.962 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.962+0000 7fc22effd700 1 -- 172.21.15.145:0/118201606 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc2380625d0 con 0x7fc2240a2890 2024-03-28T18:37:41.962 INFO:teuthology.orchestra.run.smithi145.stdout:Backend: cephadm 2024-03-28T18:37:41.962 INFO:teuthology.orchestra.run.smithi145.stdout:Available: Yes 2024-03-28T18:37:41.962 INFO:teuthology.orchestra.run.smithi145.stdout:Paused: No 2024-03-28T18:37:41.964 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 msgr2=0x7fc2240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:41.964 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 0x7fc2240a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc238105470 tx=0x7fc234009260 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 msgr2=0x7fc23819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc228000c00 tx=0x7fc22800bee0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 shutdown_connections 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2240a2890 0x7fc2240a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc238105fb0 0x7fc2381a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc238104410 0x7fc23819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 --2- 172.21.15.145:0/118201606 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc238105610 0x7fc23819aaf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.964+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 >> 172.21.15.145:0/118201606 conn(0x7fc2380ff9b0 msgr2=0x7fc2381088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.965+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 shutdown_connections 2024-03-28T18:37:41.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:41.965+0000 7fc240130700 1 -- 172.21.15.145:0/118201606 wait complete. 2024-03-28T18:37:42.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T18:37:42.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:42 smithi192 ceph-mon[36589]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:42.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:42 smithi164 ceph-mon[36248]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:43.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:42 smithi145 ceph-mon[32151]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:43.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:43 smithi145 ceph-mon[32151]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:43.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:43 smithi192 ceph-mon[36589]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:43.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:43 smithi164 ceph-mon[36248]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:44.717 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:44 smithi145 ceph-mon[32151]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:44.799 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 -- 172.21.15.145:0/2675967733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 msgr2=0x7fd474100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2675967733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd474100e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd4680098f0 tx=0x7fd468009c00 comp rx=0 tx=0).stop 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 -- 172.21.15.145:0/2675967733 shutdown_connections 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2675967733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd474101d60 0x7fd4741021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2675967733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd474100e60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2675967733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4741013a0 0x7fd474101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 -- 172.21.15.145:0/2675967733 >> 172.21.15.145:0/2675967733 conn(0x7fd4740fc660 msgr2=0x7fd4740feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 -- 172.21.15.145:0/2675967733 shutdown_connections 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 -- 172.21.15.145:0/2675967733 wait complete. 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.712+0000 7fd47aabd700 1 Processor -- start 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 -- start start 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4741013a0 0x7fd47419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 0x7fd4741a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:44.800 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47410ffc0 con 0x7fd4741013a0 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47410fe40 con 0x7fd474101d60 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47aabd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474110140 con 0x7fd474100a40 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd473fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4741013a0 0x7fd47419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57180/0 (socket says 172.21.15.145:57180) 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 -- 172.21.15.145:0/2267257257 learned_addr learned my addr 172.21.15.145:0/2267257257 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47905a700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 0x7fd4741a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 -- 172.21.15.145:0/2267257257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 msgr2=0x7fd4741a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 0x7fd4741a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 -- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4741013a0 msgr2=0x7fd47419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4741013a0 0x7fd47419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 -- 172.21.15.145:0/2267257257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4741a2300 con 0x7fd474100a40 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd47905a700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 0x7fd4741a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.713+0000 7fd478859700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd468010ff0 tx=0x7fd468005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.714+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468025070 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.714+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd468005520 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.714+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468003ce0 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.714+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4741a2590 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.714+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4741a29f0 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.716+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd468028070 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.716+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474068e50 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.717+0000 7fd471ffb700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 0x7fd4640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.717+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd4680cf5b0 con 0x7fd474100a40 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.717+0000 7fd473fff700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 0x7fd4640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.718+0000 7fd473fff700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 0x7fd4640a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd460009c60 tx=0x7fd460009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.721+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd46805c740 con 0x7fd474100a40 2024-03-28T18:37:44.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:44 smithi192 ceph-mon[36589]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:44.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.931+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4740625d0 con 0x7fd4640a2910 2024-03-28T18:37:44.948 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.948+0000 7fd471ffb700 1 -- 172.21.15.145:0/2267257257 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4092 (secure 0 0 0) 0x7fd4740625d0 con 0x7fd4640a2910 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.a smithi145 running (2m) 21s ago 2m 18.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f9a899a7e110 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.b smithi164 running (2m) 54s ago 2m 16.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d55e683a26f5 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.c smithi192 running (2m) 54s ago 2m 18.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2d0caec4a3c6 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.d smithi145 running (2m) 21s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 20bba0ebfaf6 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.e smithi164 running (2m) 54s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3e7519f502a3 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.f smithi192 running (2m) 54s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2d59c9eb3b67 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.g smithi145 running (2m) 21s ago 2m 16.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bf97af9657e4 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.h smithi164 running (2m) 54s ago 2m 18.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d3238ae76da1 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.i smithi192 running (2m) 54s ago 2m 26.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 6c1f0d8f7f82 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.j smithi145 running (2m) 21s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 189b793e27aa 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.k smithi164 running (2m) 54s ago 2m 17.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 04156e0f2135 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mds.l smithi192 running (2m) 54s ago 2m 18.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 863eedda497a 2024-03-28T18:37:44.949 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.x smithi145 *:9283,8765 running (9m) 21s ago 9m 514M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f599c4e55841 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.y smithi164 *:8443,8765 running (7m) 54s ago 7m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 251d73a10f39 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:mgr.z smithi192 *:8443,8765 running (7m) 54s ago 7m 453M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3e659b1275d6 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:mon.a smithi145 running (9m) 21s ago 9m 50.0M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 01c03d436a16 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:mon.b smithi164 running (8m) 54s ago 8m 42.2M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a2f122f90cc3 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:mon.c smithi192 running (8m) 54s ago 8m 39.4M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b85af841afcb 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 smithi145 running (7m) 21s ago 7m 46.4M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e1140b631737 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 smithi164 running (7m) 54s ago 7m 58.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 95f2b6ac42b8 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 smithi192 running (6m) 54s ago 6m 47.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7812b7a2ca01 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 smithi145 running (6m) 21s ago 6m 47.0M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c6cdcedd03a7 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 smithi164 running (5m) 54s ago 5m 53.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d5bc7f950bbf 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 smithi192 running (5m) 54s ago 5m 55.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c0fdfbd31606 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 smithi145 running (4m) 21s ago 4m 42.3M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 90e78dd2ccb5 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 smithi164 running (4m) 54s ago 4m 53.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 69946ceb4e2a 2024-03-28T18:37:44.950 INFO:teuthology.orchestra.run.smithi145.stdout:osd.8 smithi192 running (4m) 54s ago 4m 55.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5c7fd4b54cb0 2024-03-28T18:37:44.951 INFO:teuthology.orchestra.run.smithi145.stdout:osd.9 smithi145 running (3m) 21s ago 3m 55.4M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d44955764a8c 2024-03-28T18:37:44.951 INFO:teuthology.orchestra.run.smithi145.stdout:osd.10 smithi164 running (3m) 54s ago 3m 54.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d193f7bb90fa 2024-03-28T18:37:44.951 INFO:teuthology.orchestra.run.smithi145.stdout:osd.11 smithi192 running (2m) 54s ago 2m 44.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a60188623b97 2024-03-28T18:37:44.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 msgr2=0x7fd4640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 0x7fd4640a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd460009c60 tx=0x7fd460009260 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 msgr2=0x7fd47419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd468010ff0 tx=0x7fd468005040 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 shutdown_connections 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd4640a2910 0x7fd4640a4dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd474101d60 0x7fd4741a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd474100a40 0x7fd47419a6a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 --2- 172.21.15.145:0/2267257257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4741013a0 0x7fd47419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 >> 172.21.15.145:0/2267257257 conn(0x7fd4740fc660 msgr2=0x7fd4740ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 shutdown_connections 2024-03-28T18:37:44.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:44.951+0000 7fd47aabd700 1 -- 172.21.15.145:0/2267257257 wait complete. 2024-03-28T18:37:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:44 smithi164 ceph-mon[36248]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:45.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T18:37:46.049 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:46.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:46 smithi145 ceph-mon[32151]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:46.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:46 smithi145 ceph-mon[32151]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:46.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:46 smithi192 ceph-mon[36589]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:46.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:46 smithi192 ceph-mon[36589]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:46 smithi164 ceph-mon[36248]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:46 smithi164 ceph-mon[36248]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.811+0000 7fba08d1c700 1 -- 172.21.15.145:0/1010205844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 msgr2=0x7fba04107960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.811+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1010205844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba04107960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009990 tx=0x7fb9f8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 -- 172.21.15.145:0/1010205844 shutdown_connections 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1010205844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0410a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1010205844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba04105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1010205844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba04107960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.825 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 -- 172.21.15.145:0/1010205844 >> 172.21.15.145:0/1010205844 conn(0x7fba040fc650 msgr2=0x7fba040fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 -- 172.21.15.145:0/1010205844 shutdown_connections 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.812+0000 7fba08d1c700 1 -- 172.21.15.145:0/1010205844 wait complete. 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 Processor -- start 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 -- start start 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba041983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0419f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0410ddc0 con 0x7fba04105570 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0410dc40 con 0x7fba04107ea0 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba08d1c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0410df40 con 0x7fba04102c40 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba0259c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba041983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43762/0 (socket says 172.21.15.145:43762) 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 -- 172.21.15.145:0/1514606352 learned_addr learned my addr 172.21.15.145:0/1514606352 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba02d9d700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0419f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 -- 172.21.15.145:0/1514606352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 msgr2=0x7fba041983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba041983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 -- 172.21.15.145:0/1514606352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 msgr2=0x7fba0419f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0419f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 -- 172.21.15.145:0/1514606352 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba041a0000 con 0x7fba04105570 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba02d9d700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba0259c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba041983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.813+0000 7fba01d9b700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800bbb0 tx=0x7fb9f800bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.814+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8026070 con 0x7fba04105570 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.814+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9f800f460 con 0x7fba04105570 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.814+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba041a0290 con 0x7fba04105570 2024-03-28T18:37:47.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.814+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8021670 con 0x7fba04105570 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.814+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba041a06e0 con 0x7fba04105570 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.815+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba04068e50 con 0x7fba04105570 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.816+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb9f8029070 con 0x7fba04105570 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.816+0000 7fb9f37fe700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 0x7fb9ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.816+0000 7fba0259c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 0x7fb9ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:47.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.816+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb9f80cfcd0 con 0x7fba04105570 2024-03-28T18:37:47.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.817+0000 7fba0259c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 0x7fb9ec0a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4011490 tx=0x7fb9f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:47.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:47.821+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9f805d740 con 0x7fba04105570 2024-03-28T18:37:48.119 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.119+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fba040625d0 con 0x7fb9ec0a2890 2024-03-28T18:37:48.124 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.125+0000 7fb9f37fe700 1 -- 172.21.15.145:0/1514606352 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fba040625d0 con 0x7fb9ec0a2890 2024-03-28T18:37:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T18:37:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:mds.all 12/12 57s ago 2m smithi145=a;smithi145=d;smithi145=g;smithi145=j;smithi164=b;smithi164=e;smithi164=h;smithi164=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-28T18:37:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:mgr 3/3 57s ago 7m smithi145=x;smithi164=y;smithi192=z;count:3 2024-03-28T18:37:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:mon 3/3 57s ago 8m smithi145:172.21.15.145=a;smithi164:172.21.15.164=b;smithi192:172.21.15.192=c;count:3 2024-03-28T18:37:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:osd 12 57s ago - 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 msgr2=0x7fb9ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 0x7fb9ec0a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4011490 tx=0x7fb9f4009260 comp rx=0 tx=0).stop 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 msgr2=0x7fba041988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800bbb0 tx=0x7fb9f800bbe0 comp rx=0 tx=0).stop 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 shutdown_connections 2024-03-28T18:37:48.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb9ec0a2890 0x7fb9ec0a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba04107ea0 0x7fba0419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba04102c40 0x7fba041983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 --2- 172.21.15.145:0/1514606352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba04105570 0x7fba041988e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 >> 172.21.15.145:0/1514606352 conn(0x7fba040fc650 msgr2=0x7fba04103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 shutdown_connections 2024-03-28T18:37:48.128 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:48.127+0000 7fba08d1c700 1 -- 172.21.15.145:0/1514606352 wait complete. 2024-03-28T18:37:48.898 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:48 smithi145 ceph-mon[32151]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:48.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:48 smithi192 ceph-mon[36589]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:48.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T18:37:48.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:48 smithi164 ceph-mon[36248]: pgmap v350: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:37:49.183 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:49.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:49 smithi192 ceph-mon[36589]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:49.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:49 smithi164 ceph-mon[36248]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:50.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:49 smithi145 ceph-mon[32151]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:50.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:50 smithi192 ceph-mon[36589]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:50 smithi164 ceph-mon[36248]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:51.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:50 smithi145 ceph-mon[32151]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:37:51.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 -- 172.21.15.145:0/3207774915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d301055e0 msgr2=0x7f3d301079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:51.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 --2- 172.21.15.145:0/3207774915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d301055e0 0x7f3d301079d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009990 tx=0x7f3d24009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 -- 172.21.15.145:0/3207774915 shutdown_connections 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 --2- 172.21.15.145:0/3207774915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30107f10 0x7f3d3010a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 --2- 172.21.15.145:0/3207774915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30102cb0 0x7f3d301050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 --2- 172.21.15.145:0/3207774915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d301055e0 0x7f3d301079d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 -- 172.21.15.145:0/3207774915 >> 172.21.15.145:0/3207774915 conn(0x7f3d300fc660 msgr2=0x7f3d300feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 -- 172.21.15.145:0/3207774915 shutdown_connections 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.130+0000 7f3d360c3700 1 -- 172.21.15.145:0/3207774915 wait complete. 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 Processor -- start 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 -- start start 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301055e0 0x7f3d30198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 0x7f3d3019f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010ddd0 con 0x7f3d301055e0 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010dc50 con 0x7f3d30107f10 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d360c3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010df50 con 0x7f3d30102cb0 2024-03-28T18:37:51.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d2effd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301055e0 0x7f3d30198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d2f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d2f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57208/0 (socket says 172.21.15.145:57208) 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d2ffff700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 0x7f3d3019f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.131+0000 7f3d2f7fe700 1 -- 172.21.15.145:0/1812833390 learned_addr learned my addr 172.21.15.145:0/1812833390 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 -- 172.21.15.145:0/1812833390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 msgr2=0x7f3d3019f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 0x7f3d3019f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 -- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301055e0 msgr2=0x7f3d30198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301055e0 0x7f3d30198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 -- 172.21.15.145:0/1812833390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d301a0080 con 0x7f3d30102cb0 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2ffff700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 0x7f3d3019f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:51.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2f7fe700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e9f0 tx=0x7f3d2000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2000cbd0 con 0x7f3d30102cb0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d2000ee90 con 0x7f3d30102cb0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d301a0370 con 0x7f3d30102cb0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d200056c0 con 0x7f3d30102cb0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.132+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301a08b0 con 0x7f3d30102cb0 2024-03-28T18:37:51.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.133+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d30068e50 con 0x7f3d30102cb0 2024-03-28T18:37:51.135 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.135+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3d20044910 con 0x7f3d30102cb0 2024-03-28T18:37:51.135 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.135+0000 7f3d2cff9700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 0x7f3d180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:51.135 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.135+0000 7f3d2effd700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 0x7f3d180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:51.135 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.135+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3d200cefe0 con 0x7f3d30102cb0 2024-03-28T18:37:51.136 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.136+0000 7f3d2effd700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 0x7f3d180a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b7b0 tx=0x7f3d2400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:51.138 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.138+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d200203a0 con 0x7f3d30102cb0 2024-03-28T18:37:51.415 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.415+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d300625d0 con 0x7f3d180a2910 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.417+0000 7f3d2cff9700 1 -- 172.21.15.145:0/1812833390 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f3d300625d0 con 0x7f3d180a2910 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stdout:HOST ADDR LABELS STATUS 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145 172.21.15.145 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stdout:smithi164 172.21.15.164 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stdout:smithi192 172.21.15.192 2024-03-28T18:37:51.417 INFO:teuthology.orchestra.run.smithi145.stdout:3 hosts in cluster 2024-03-28T18:37:51.419 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.419+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 msgr2=0x7f3d180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.419+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 0x7f3d180a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b7b0 tx=0x7f3d2400b880 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.419+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 msgr2=0x7f3d30198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.419+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e9f0 tx=0x7f3d2000ed00 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 shutdown_connections 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3d180a2910 0x7f3d180a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d30107f10 0x7f3d3019f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d30102cb0 0x7f3d30198420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 --2- 172.21.15.145:0/1812833390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d301055e0 0x7f3d30198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 >> 172.21.15.145:0/1812833390 conn(0x7f3d300fc660 msgr2=0x7f3d30103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 shutdown_connections 2024-03-28T18:37:51.420 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:51.420+0000 7f3d360c3700 1 -- 172.21.15.145:0/1812833390 wait complete. 2024-03-28T18:37:52.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T18:37:52.432 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:52.805 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:52 smithi145 ceph-mon[32151]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:52.805 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:52 smithi145 ceph-mon[32151]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:52.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:52 smithi192 ceph-mon[36589]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:52.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:52 smithi192 ceph-mon[36589]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:52 smithi164 ceph-mon[36248]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:52 smithi164 ceph-mon[36248]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:37:53.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T18:37:53.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-28T18:37:53.886 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:53 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-28T18:37:53.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T18:37:53.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-28T18:37:53.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:53 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-28T18:37:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T18:37:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-03-28T18:37:53.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:53 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-28T18:37:54.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 -- 172.21.15.145:0/2036108196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84341022c0 msgr2=0x7f8434102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:54.282 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 --2- 172.21.15.145:0/2036108196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84341022c0 0x7f8434102720 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f84240099c0 tx=0x7f8424009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 -- 172.21.15.145:0/2036108196 shutdown_connections 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 --2- 172.21.15.145:0/2036108196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8434102c60 0x7f8434109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 --2- 172.21.15.145:0/2036108196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341010c0 0x7f84341014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 --2- 172.21.15.145:0/2036108196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84341022c0 0x7f8434102720 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 -- 172.21.15.145:0/2036108196 >> 172.21.15.145:0/2036108196 conn(0x7f84340fc660 msgr2=0x7f84340feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 -- 172.21.15.145:0/2036108196 shutdown_connections 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.276+0000 7f843c522700 1 -- 172.21.15.145:0/2036108196 wait complete. 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 Processor -- start 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 -- start start 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 0x7f843419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:54.283 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843410cca0 con 0x7f8434102c60 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843410cb20 con 0x7f84341010c0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.277+0000 7f843c522700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843410ce20 con 0x7f84341022c0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843a2be700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843aabf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 0x7f843419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843a2be700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:49336/0 (socket says 172.21.15.145:49336) 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843a2be700 1 -- 172.21.15.145:0/4190421828 learned_addr learned my addr 172.21.15.145:0/4190421828 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:55010/0 (socket says 172.21.15.145:55010) 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 -- 172.21.15.145:0/4190421828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 msgr2=0x7f843406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 -- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 msgr2=0x7f843419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 0x7f843419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 -- 172.21.15.145:0/4190421828 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84341a0080 con 0x7f84341022c0 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843aabf700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 0x7f843419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:54.284 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843a2be700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f8439abd700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f842400b580 tx=0x7f842400f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424009e60 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84341a0310 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8424003e40 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424005490 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.278+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84341a07f0 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.279+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8434068e50 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.281+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f842400fab0 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.282+0000 7f842b7fe700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 0x7f84200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.282+0000 7f843a2be700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 0x7f84200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.282+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f84240cfc90 con 0x7f84341022c0 2024-03-28T18:37:54.285 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.282+0000 7f843a2be700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 0x7f84200a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8430005950 tx=0x7f8430009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:54.286 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.284+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f842405d740 con 0x7f84341022c0 2024-03-28T18:37:54.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.495+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f84340625d0 con 0x7f84200a2890 2024-03-28T18:37:54.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.497+0000 7f842b7fe700 1 -- 172.21.15.145:0/4190421828 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f84340625d0 con 0x7f84200a2890 2024-03-28T18:37:54.497 INFO:teuthology.orchestra.run.smithi145.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T18:37:54.497 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N9400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:37:54.497 INFO:teuthology.orchestra.run.smithi145.stdout:smithi164 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300H0400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:37:54.497 INFO:teuthology.orchestra.run.smithi145.stdout:smithi192 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500NN400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 msgr2=0x7f84200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 0x7f84200a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8430005950 tx=0x7f8430009260 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 msgr2=0x7f843419d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f842400b580 tx=0x7f842400f710 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 shutdown_connections 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f84200a2890 0x7f84200a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8434102c60 0x7f843419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84341010c0 0x7f843406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 --2- 172.21.15.145:0/4190421828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84341022c0 0x7f843419d170 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 >> 172.21.15.145:0/4190421828 conn(0x7f84340fc660 msgr2=0x7f84341056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 shutdown_connections 2024-03-28T18:37:54.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:54.500+0000 7f843c522700 1 -- 172.21.15.145:0/4190421828 wait complete. 2024-03-28T18:37:54.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:54 smithi145 ceph-mon[32151]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:54.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T18:37:54.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-28T18:37:54.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-28T18:37:54.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:54 smithi145 ceph-mon[32151]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:37:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:54 smithi192 ceph-mon[36589]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T18:37:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-28T18:37:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-28T18:37:54.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:54 smithi192 ceph-mon[36589]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:37:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:54 smithi164 ceph-mon[36248]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:37:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T18:37:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-03-28T18:37:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-28T18:37:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:54 smithi164 ceph-mon[36248]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:37:55.242 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:37:55.254 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:37:55.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T18:37:55.500 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:55.890 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:55 smithi145 ceph-mon[32151]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:55.890 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:55 smithi145 ceph-mon[32151]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:37:55.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:55 smithi192 ceph-mon[36589]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:55.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:55 smithi192 ceph-mon[36589]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:37:55.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:55 smithi164 ceph-mon[36248]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:55.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:55 smithi164 ceph-mon[36248]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:37:56.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:56 smithi192 ceph-mon[36589]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T18:37:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:56 smithi164 ceph-mon[36248]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T18:37:56.979 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:56 smithi145 ceph-mon[32151]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.212+0000 7f64f88cb700 1 -- 172.21.15.145:0/521769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 msgr2=0x7f64f4102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.212+0000 7f64f88cb700 1 --2- 172.21.15.145:0/521769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f4102720 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f64e80099c0 tx=0x7f64e8009cd0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 -- 172.21.15.145:0/521769362 shutdown_connections 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 --2- 172.21.15.145:0/521769362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f4102c60 0x7f64f4109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 --2- 172.21.15.145:0/521769362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f41010c0 0x7f64f41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 --2- 172.21.15.145:0/521769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f4102720 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 -- 172.21.15.145:0/521769362 >> 172.21.15.145:0/521769362 conn(0x7f64f40fc660 msgr2=0x7f64f40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 -- 172.21.15.145:0/521769362 shutdown_connections 2024-03-28T18:37:57.213 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 -- 172.21.15.145:0/521769362 wait complete. 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 Processor -- start 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.213+0000 7f64f88cb700 1 -- start start 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 0x7f64f419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f4102c60 0x7f64f41a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f410cc80 con 0x7f64f41022c0 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f410cb00 con 0x7f64f41010c0 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f88cb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f410ce00 con 0x7f64f4102c60 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f1d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:57.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f1d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53912/0 (socket says 172.21.15.145:53912) 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 0x7f64f419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f1d9b700 1 -- 172.21.15.145:0/2903129455 learned_addr learned my addr 172.21.15.145:0/2903129455 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f2d9d700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f4102c60 0x7f64f41a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 -- 172.21.15.145:0/2903129455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f4102c60 msgr2=0x7f64f41a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f4102c60 0x7f64f41a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 -- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 msgr2=0x7f64f419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 -- 172.21.15.145:0/2903129455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f41a2280 con 0x7f64f41010c0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f1d9b700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64f259c700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 0x7f64f419a620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e9c0 tx=0x7f64e400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.214+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e400cb70 con 0x7f64f41010c0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.215+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f41a2570 con 0x7f64f41010c0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.215+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64e400ee60 con 0x7f64f41010c0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.215+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4018660 con 0x7f64f41010c0 2024-03-28T18:37:57.216 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.215+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f41a2b00 con 0x7f64f41010c0 2024-03-28T18:37:57.217 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.215+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f4068e50 con 0x7f64f41010c0 2024-03-28T18:37:57.217 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.217+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f64e401f020 con 0x7f64f41010c0 2024-03-28T18:37:57.217 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.217+0000 7f64e37fe700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 0x7f64dc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:57.218 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.217+0000 7f64f1d9b700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 0x7f64dc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:57.218 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.217+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f64e40cf250 con 0x7f64f41010c0 2024-03-28T18:37:57.218 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.218+0000 7f64f1d9b700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 0x7f64dc0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009990 tx=0x7f64e8003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:57.225 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.225+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f64e409bee0 con 0x7f64f41010c0 2024-03-28T18:37:57.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.472+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f64f41a2f40 con 0x7f64f41010c0 2024-03-28T18:37:57.472 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.473+0000 7f64e37fe700 1 -- 172.21.15.145:0/2903129455 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+3052 (secure 0 0 0) 0x7f64e409b7e0 con 0x7f64f41010c0 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:e28 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:legacy client fscid: 1 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:Filesystem 'cephfs' (1) 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:fs_name cephfs 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:epoch 28 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:created 2024-03-28T18:35:50.266038+0000 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:modified 2024-03-28T18:36:13.026843+0000 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:tableserver 0 2024-03-28T18:37:57.473 INFO:teuthology.orchestra.run.smithi145.stdout:root 0 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:session_timeout 300 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:session_autoclose 300 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:max_file_size 1099511627776 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:max_xattr_size 65536 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:required_client_features {} 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:last_failure 0 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:last_failure_osd_epoch 0 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:max_mds 5 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:in 0,1,2,3,4 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:up {0=24496,1=24491,2=24469,3=24461,4=14631} 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:failed 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:damaged 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:stopped 2024-03-28T18:37:57.474 INFO:teuthology.orchestra.run.smithi145.stdout:data_pools [3,4] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:metadata_pool 2 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:inline_data disabled 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:balancer 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:bal_rank_mask -1 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:standby_count_wanted 0 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:qdb_cluster leader: 24496 members: 24469,24491,14631,24461,24496 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.i{0:24496} state up:active seq 5 addr [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.f{0:24457} state up:standby-replay seq 1 addr [v2:172.21.15.192:6834/1226474966,v1:172.21.15.192:6835/1226474966] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b{1:24491} state up:active seq 8 addr [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.e{1:24431} state up:standby-replay seq 1 addr [v2:172.21.15.164:6834/3793155920,v1:172.21.15.164:6835/3793155920] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c{2:24469} state up:active seq 12 addr [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.l{2:24415} state up:standby-replay seq 1 addr [v2:172.21.15.192:6832/3704007459,v1:172.21.15.192:6833/3704007459] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.h{3:24461} state up:active seq 11 addr [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.475 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.k{3:24395} state up:standby-replay seq 1 addr [v2:172.21.15.164:6832/3835667312,v1:172.21.15.164:6833/3835667312] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a{4:14631} state up:active seq 9 addr [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.g{4:14619} state up:standby-replay seq 1 addr [v2:172.21.15.145:6838/1703430127,v1:172.21.15.145:6839/1703430127] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout:Standby daemons: 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.145:6834/1607345926,v1:172.21.15.145:6835/1607345926] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.476 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.145:6836/3754178140,v1:172.21.15.145:6837/3754178140] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.475+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 msgr2=0x7f64dc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.475+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 0x7f64dc0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009990 tx=0x7f64e8003860 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.475+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 msgr2=0x7f64f419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.475+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 0x7f64f419a620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e9c0 tx=0x7f64e400ecd0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 shutdown_connections 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f64dc0a2890 0x7f64dc0a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64f4102c60 0x7f64f41a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64f41010c0 0x7f64f419a620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 --2- 172.21.15.145:0/2903129455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f41022c0 0x7f64f419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 >> 172.21.15.145:0/2903129455 conn(0x7f64f40fc660 msgr2=0x7f64f40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 shutdown_connections 2024-03-28T18:37:57.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:37:57.476+0000 7f64f88cb700 1 -- 172.21.15.145:0/2903129455 wait complete. 2024-03-28T18:37:57.478 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:37:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:57 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2903129455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:37:57.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:57 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2903129455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:37:57.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:57 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2903129455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:37:58.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T18:37:58.467 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:58.894 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:37:58 smithi145 ceph-mon[32151]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T18:37:58.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:37:58 smithi192 ceph-mon[36589]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T18:37:58.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:37:58 smithi164 ceph-mon[36248]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.240+0000 7f77d83d1700 1 -- 172.21.15.145:0/3350513045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0102c50 msgr2=0x7f77d0105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.240+0000 7f77d83d1700 1 --2- 172.21.15.145:0/3350513045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0102c50 0x7f77d0105040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009990 tx=0x7f77c8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- 172.21.15.145:0/3350513045 shutdown_connections 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- 172.21.15.145:0/3350513045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0107eb0 0x7f77d010a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- 172.21.15.145:0/3350513045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0102c50 0x7f77d0105040 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- 172.21.15.145:0/3350513045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0105580 0x7f77d0107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- 172.21.15.145:0/3350513045 >> 172.21.15.145:0/3350513045 conn(0x7f77d00fc660 msgr2=0x7f77d00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- 172.21.15.145:0/3350513045 shutdown_connections 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- 172.21.15.145:0/3350513045 wait complete. 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 Processor -- start 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- start start 2024-03-28T18:38:00.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 0x7f77d0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 0x7f77d0198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0107eb0 0x7f77d019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d010ddf0 con 0x7f77d0105580 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d010dc70 con 0x7f77d0102c50 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d83d1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d010df70 con 0x7f77d0107eb0 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d596c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 0x7f77d0198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d616d700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 0x7f77d0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d596c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 0x7f77d0198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53936/0 (socket says 172.21.15.145:53936) 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.241+0000 7f77d596c700 1 -- 172.21.15.145:0/708312349 learned_addr learned my addr 172.21.15.145:0/708312349 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d696e700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0107eb0 0x7f77d019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 -- 172.21.15.145:0/708312349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0107eb0 msgr2=0x7f77d019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0107eb0 0x7f77d019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 -- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 msgr2=0x7f77d0198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 0x7f77d0198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 -- 172.21.15.145:0/708312349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d01a00b0 con 0x7f77d0102c50 2024-03-28T18:38:00.247 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d616d700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 0x7f77d0198450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098e0 tx=0x7f77c800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8026070 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d01a0340 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c8003d70 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8021410 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.242+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d01a07a0 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.243+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d0068e50 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.244+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f77c8029070 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.245+0000 7f77c77fe700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 0x7f77bc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.245+0000 7f77d596c700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 0x7f77bc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.245+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f77c80cfb90 con 0x7f77d0102c50 2024-03-28T18:38:00.248 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.245+0000 7f77d596c700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 0x7f77bc0a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009d60 tx=0x7f77c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:00.249 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.249+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f77c805d740 con 0x7f77d0102c50 2024-03-28T18:38:00.450 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.450+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f77d0062da0 con 0x7f77d0102c50 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.451+0000 7f77c77fe700 1 -- 172.21.15.145:0/708312349 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) v1 ==== 56+0+5703 (secure 0 0 0) 0x7f77c8013070 con 0x7f77d0102c50 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:epoch 95 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:fsid b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:created 2024-03-28T18:28:01.180162+0000 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:modified 2024-03-28T18:37:54.633430+0000 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:crush_version 27 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:full_ratio 0.95 2024-03-28T18:38:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:backfillfull_ratio 0.9 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:nearfull_ratio 0.85 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:require_min_compat_client luminous 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:min_compat_client luminous 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:require_osd_release squid 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:stretch_mode_enabled false 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:max_osd 12 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6802/2900391364,v1:172.21.15.145:6803/2900391364] [v2:172.21.15.145:6804/2900391364,v1:172.21.15.145:6805/2900391364] exists,up d20b632f-9b4a-4f94-82e3-3b5668acd7f0 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6800/3569377863,v1:172.21.15.164:6801/3569377863] [v2:172.21.15.164:6802/3569377863,v1:172.21.15.164:6803/3569377863] exists,up 70cfca66-0490-4c8f-9d54-0a64f05392a0 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/629223859,v1:172.21.15.192:6801/629223859] [v2:172.21.15.192:6802/629223859,v1:172.21.15.192:6803/629223859] exists,up 36e9e51d-79a5-402a-bf8b-9bb427e8e847 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.3 up in weight 1 up_from 25 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6810/1910028611,v1:172.21.15.145:6811/1910028611] [v2:172.21.15.145:6812/1910028611,v1:172.21.15.145:6813/1910028611] exists,up 752a99f2-3b2f-4964-8e7c-1fd521d16fff 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6808/2404350787,v1:172.21.15.164:6809/2404350787] [v2:172.21.15.164:6810/2404350787,v1:172.21.15.164:6811/2404350787] exists,up fea7e9f1-4577-439a-81a8-5b6bf2e810e6 2024-03-28T18:38:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:osd.5 up in weight 1 up_from 39 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/22255675,v1:172.21.15.192:6809/22255675] [v2:172.21.15.192:6810/22255675,v1:172.21.15.192:6811/22255675] exists,up f883f88f-77ee-4a4e-9d4e-b7f0469c50fc 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.6 up in weight 1 up_from 47 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6818/2385620152,v1:172.21.15.145:6819/2385620152] [v2:172.21.15.145:6820/2385620152,v1:172.21.15.145:6821/2385620152] exists,up bf2bf55a-48f0-46b4-acbd-f8c012e9b39d 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.7 up in weight 1 up_from 53 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6816/795549152,v1:172.21.15.164:6817/795549152] [v2:172.21.15.164:6818/795549152,v1:172.21.15.164:6819/795549152] exists,up 532710ac-62e7-479f-885d-5604af92fd37 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.8 up in weight 1 up_from 61 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/1092086458,v1:172.21.15.192:6817/1092086458] [v2:172.21.15.192:6818/1092086458,v1:172.21.15.192:6819/1092086458] exists,up dc3ae89b-0840-440c-8bce-854d680e9538 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.9 up in weight 1 up_from 69 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6826/2302293093,v1:172.21.15.145:6827/2302293093] [v2:172.21.15.145:6828/2302293093,v1:172.21.15.145:6829/2302293093] exists,up 5c299158-d952-43b7-8ef1-6da9ab558f86 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.10 up in weight 1 up_from 74 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6824/3964117002,v1:172.21.15.164:6825/3964117002] [v2:172.21.15.164:6826/3964117002,v1:172.21.15.164:6827/3964117002] exists,up bce4fe04-fb9d-4108-ab8f-420cae8243ed 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:osd.11 up in weight 1 up_from 82 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/3207430647,v1:172.21.15.192:6825/3207430647] [v2:172.21.15.192:6826/3207430647,v1:172.21.15.192:6827/3207430647] exists,up d95d0633-327c-4fe9-a150-07eaa70cbcec 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 2.0 [7,11] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 2.c [2,1] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 2.10 [8,11] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 2.16 [2,4] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 3.c [3,9] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 3.11 [5,9] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 3.27 [4,8] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 3.34 [4,9] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 3.3a [4,6] 2024-03-28T18:38:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 4.11 [5,8] 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 4.12 [9,10] 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 4.1b [9,8] 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:pg_upmap_items 4.1f [9,8] 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/606368710 expires 2024-03-29T18:28:53.394017+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/4006486926 expires 2024-03-29T18:28:53.394017+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/2249018729 expires 2024-03-29T18:28:28.346767+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:6801/2044834522 expires 2024-03-29T18:28:28.346767+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:6800/2089479814 expires 2024-03-29T18:28:53.394017+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/3480949122 expires 2024-03-29T18:28:28.346767+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/832150839 expires 2024-03-29T18:28:28.346767+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:6800/2044834522 expires 2024-03-29T18:28:28.346767+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:6801/2089479814 expires 2024-03-29T18:28:53.394017+0000 2024-03-28T18:38:00.454 INFO:teuthology.orchestra.run.smithi145.stdout:blocklist 172.21.15.145:0/3638297666 expires 2024-03-29T18:28:53.394017+0000 2024-03-28T18:38:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 msgr2=0x7f77bc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 0x7f77bc0a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009d60 tx=0x7f77c0009260 comp rx=0 tx=0).stop 2024-03-28T18:38:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 msgr2=0x7f77d0198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 0x7f77d0198450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77c80098e0 tx=0x7f77c800bbb0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 shutdown_connections 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f77bc0a2890 0x7f77bc0a4d50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77d0107eb0 0x7f77d019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77d0102c50 0x7f77d0198450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 --2- 172.21.15.145:0/708312349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77d0105580 0x7f77d0198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 >> 172.21.15.145:0/708312349 conn(0x7f77d00fc660 msgr2=0x7f77d0103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 shutdown_connections 2024-03-28T18:38:00.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:00.454+0000 7f77d83d1700 1 -- 172.21.15.145:0/708312349 wait complete. 2024-03-28T18:38:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:00 smithi145 ceph-mon[32151]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-28T18:38:00.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:00 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/708312349' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:38:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:00 smithi192 ceph-mon[36589]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-28T18:38:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:00 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/708312349' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:38:00.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:00 smithi164 ceph-mon[36248]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-28T18:38:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:00 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/708312349' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:38:01.207 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T18:38:01.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:38:01.333 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 -- 172.21.15.145:0/3260414500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc0ff9b0 msgr2=0x7f0fcc0ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.333 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3260414500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc0ffdd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009930 tx=0x7f0fbc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:01.333 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 -- 172.21.15.145:0/3260414500 shutdown_connections 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3260414500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc103340 0x7f0fcc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3260414500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc102980 0x7f0fcc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3260414500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc0ffdd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 -- 172.21.15.145:0/3260414500 >> 172.21.15.145:0/3260414500 conn(0x7f0fcc0fb2e0 msgr2=0x7f0fcc0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 -- 172.21.15.145:0/3260414500 shutdown_connections 2024-03-28T18:38:01.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.333+0000 7f0fd2562640 1 -- 172.21.15.145:0/3260414500 wait complete. 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 Processor -- start 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 -- start start 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc103340 0x7f0fcc19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10d360 con 0x7f0fcc102980 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10d1e0 con 0x7f0fcc103340 2024-03-28T18:38:01.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fd2562640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10d4e0 con 0x7f0fcc0ff9b0 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fcb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fcb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53956/0 (socket says 172.21.15.145:53956) 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fcb7fe640 1 -- 172.21.15.145:0/3114279970 learned_addr learned my addr 172.21.15.145:0/3114279970 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.334+0000 7f0fcbfff640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fd0aff640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc103340 0x7f0fcc19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 -- 172.21.15.145:0/3114279970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc0ff9b0 msgr2=0x7f0fcc197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 -- 172.21.15.145:0/3114279970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc103340 msgr2=0x7f0fcc19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc103340 0x7f0fcc19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 -- 172.21.15.145:0/3114279970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc19f680 con 0x7f0fcc102980 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fcb7fe640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e9f0 tx=0x7f0fc000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc000ccc0 con 0x7f0fcc102980 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc19f970 con 0x7f0fcc102980 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0010910 con 0x7f0fcc102980 2024-03-28T18:38:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00188a0 con 0x7f0fcc102980 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.335+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc19fec0 con 0x7f0fcc102980 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.336+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc108b80 con 0x7f0fcc102980 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.337+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0fc0010ac0 con 0x7f0fcc102980 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.338+0000 7f0fc97fa640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 0x7f0fb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.338+0000 7f0fcbfff640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 0x7f0fb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.338+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0fc00cf250 con 0x7f0fcc102980 2024-03-28T18:38:01.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.338+0000 7f0fcbfff640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 0x7f0fb40a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc02c7b0 tx=0x7f0fbc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.342+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0fc009bf40 con 0x7f0fcc102980 2024-03-28T18:38:01.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.523+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fcc0622a0 con 0x7f0fcc102980 2024-03-28T18:38:01.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.524+0000 7f0fc97fa640 1 -- 172.21.15.145:0/3114279970 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7f0fc009b840 con 0x7f0fcc102980 2024-03-28T18:38:01.524 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:38:01.525 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:38:01.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.526+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 msgr2=0x7f0fb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.526+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 0x7f0fb40a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc02c7b0 tx=0x7f0fbc000bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.526+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 msgr2=0x7f0fcc197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.526+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e9f0 tx=0x7f0fc000edf0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 shutdown_connections 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fcc103340 0x7f0fcc19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fcc102980 0x7f0fcc197fa0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0fb40a2800 0x7f0fb40a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 --2- 172.21.15.145:0/3114279970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fcc0ff9b0 0x7f0fcc197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 >> 172.21.15.145:0/3114279970 conn(0x7f0fcc0fb2e0 msgr2=0x7f0fcc105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 shutdown_connections 2024-03-28T18:38:01.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.527+0000 7f0fd2562640 1 -- 172.21.15.145:0/3114279970 wait complete. 2024-03-28T18:38:01.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:38:01.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:38:01.647 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- 172.21.15.145:0/1780133196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f01005c0 msgr2=0x7f05f01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1780133196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f01005c0 0x7f05f01009e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009860 tx=0x7f05ec02bb70 comp rx=0 tx=0).stop 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- 172.21.15.145:0/1780133196 shutdown_connections 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1780133196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f0102170 0x7f05f0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1780133196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 0x7f05f0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1780133196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f01005c0 0x7f05f01009e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- 172.21.15.145:0/1780133196 >> 172.21.15.145:0/1780133196 conn(0x7f05f00fbb50 msgr2=0x7f05f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- 172.21.15.145:0/1780133196 shutdown_connections 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- 172.21.15.145:0/1780133196 wait complete. 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 Processor -- start 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.646+0000 7f05f6a75640 1 -- start start 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 0x7f05f01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 0x7f05f0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.648 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f010c210 con 0x7f05f0102170 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f010c090 con 0x7f05f01005c0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f010c390 con 0x7f05f01017b0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f4811640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 0x7f05f01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53978/0 (socket says 172.21.15.145:53978) 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 -- 172.21.15.145:0/1722272630 learned_addr learned my addr 172.21.15.145:0/1722272630 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05e7fff640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 0x7f05f0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 -- 172.21.15.145:0/1722272630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 msgr2=0x7f05f0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 0x7f05f0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 -- 172.21.15.145:0/1722272630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 msgr2=0x7f05f01979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 0x7f05f01979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 -- 172.21.15.145:0/1722272630 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f019f600 con 0x7f05f0102170 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f4811640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 0x7f05f01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f5012640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05e800efd0 tx=0x7f05e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8003b70 con 0x7f05f0102170 2024-03-28T18:38:01.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f019f8f0 con 0x7f05f0102170 2024-03-28T18:38:01.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05e8003d10 con 0x7f05f0102170 2024-03-28T18:38:01.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.647+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80109f0 con 0x7f05f0102170 2024-03-28T18:38:01.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.648+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f019fdf0 con 0x7f05f0102170 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.649+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f01a0050 con 0x7f05f0102170 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.650+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f05e8004020 con 0x7f05f0102170 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.650+0000 7f05e5ffb640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 0x7f05d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.650+0000 7f05f4811640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 0x7f05d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.651+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f05e80cf280 con 0x7f05f0102170 2024-03-28T18:38:01.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.651+0000 7f05f4811640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 0x7f05d80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009830 tx=0x7f05ec02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.655 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.655+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f05e809bf70 con 0x7f05f0102170 2024-03-28T18:38:01.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.835+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05f01a02c0 con 0x7f05f0102170 2024-03-28T18:38:01.838 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.836+0000 7f05e5ffb640 1 -- 172.21.15.145:0/1722272630 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05e809b870 con 0x7f05f0102170 2024-03-28T18:38:01.838 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 msgr2=0x7f05d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 0x7f05d80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009830 tx=0x7f05ec02e040 comp rx=0 tx=0).stop 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 msgr2=0x7f05f019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05e800efd0 tx=0x7f05e800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 shutdown_connections 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0102170 0x7f05f019ef90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05d80a2800 0x7f05d80a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05f01017b0 0x7f05f0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 --2- 172.21.15.145:0/1722272630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f01005c0 0x7f05f01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 >> 172.21.15.145:0/1722272630 conn(0x7f05f00fbb50 msgr2=0x7f05f00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:01.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 shutdown_connections 2024-03-28T18:38:01.840 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.838+0000 7f05f6a75640 1 -- 172.21.15.145:0/1722272630 wait complete. 2024-03-28T18:38:01.848 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:01 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3114279970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:01.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:38:01.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- 172.21.15.145:0/899260679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c102140 msgr2=0x7fcb0c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 --2- 172.21.15.145:0/899260679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c102140 0x7fcb0c104530 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009900 tx=0x7fcafc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- 172.21.15.145:0/899260679 shutdown_connections 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 --2- 172.21.15.145:0/899260679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 --2- 172.21.15.145:0/899260679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 0x7fcb0c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 --2- 172.21.15.145:0/899260679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c102140 0x7fcb0c104530 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- 172.21.15.145:0/899260679 >> 172.21.15.145:0/899260679 conn(0x7fcb0c0fbb30 msgr2=0x7fcb0c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- 172.21.15.145:0/899260679 shutdown_connections 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- 172.21.15.145:0/899260679 wait complete. 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 Processor -- start 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 -- start start 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.956+0000 7fcb1224b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 0x7fcb0c195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 0x7fcb0c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c10b180 con 0x7fcb0c1073a0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c10b000 con 0x7fcb0c102140 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c10b300 con 0x7fcb0c104a70 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 0x7fcb0c195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb03fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 0x7fcb0c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54002/0 (socket says 172.21.15.145:54002) 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 -- 172.21.15.145:0/580357814 learned_addr learned my addr 172.21.15.145:0/580357814 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 0x7fcb0c195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:49424/0 (socket says 172.21.15.145:49424) 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 -- 172.21.15.145:0/580357814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 msgr2=0x7fcb0c195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 0x7fcb0c195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 -- 172.21.15.145:0/580357814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 msgr2=0x7fcb0c195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 0x7fcb0c195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 -- 172.21.15.145:0/580357814 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c19d490 con 0x7fcb0c1073a0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb0bfff640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400eb10 tx=0x7fcb0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04004b20 con 0x7fcb0c1073a0 2024-03-28T18:38:01.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb04004cc0 con 0x7fcb0c1073a0 2024-03-28T18:38:01.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04010760 con 0x7fcb0c1073a0 2024-03-28T18:38:01.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c19d780 con 0x7fcb0c1073a0 2024-03-28T18:38:01.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.957+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c19dcd0 con 0x7fcb0c1073a0 2024-03-28T18:38:01.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.958+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c0fd620 con 0x7fcb0c1073a0 2024-03-28T18:38:01.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.960+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcb04005000 con 0x7fcb0c1073a0 2024-03-28T18:38:01.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.961+0000 7fcb097fa640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 0x7fcaf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:01.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.961+0000 7fcb0b7fe640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 0x7fcaf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:01.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.961+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fcb040cf530 con 0x7fcb0c1073a0 2024-03-28T18:38:01.961 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.961+0000 7fcb0b7fe640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 0x7fcaf80a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009230 tx=0x7fcafc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:01.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:01 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3114279970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:01.963 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:01.964+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb04053740 con 0x7fcb0c1073a0 2024-03-28T18:38:02.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.141+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb0c02d070 con 0x7fcb0c1073a0 2024-03-28T18:38:02.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.141+0000 7fcb097fa640 1 -- 172.21.15.145:0/580357814 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb0409c220 con 0x7fcb0c1073a0 2024-03-28T18:38:02.141 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 msgr2=0x7fcaf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 0x7fcaf80a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009230 tx=0x7fcafc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 msgr2=0x7fcb0c19ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400eb10 tx=0x7fcb0400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 shutdown_connections 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb0c1073a0 0x7fcb0c19ce20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcaf80a2880 0x7fcaf80a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb0c104a70 0x7fcb0c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 --2- 172.21.15.145:0/580357814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c102140 0x7fcb0c195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 >> 172.21.15.145:0/580357814 conn(0x7fcb0c0fbb30 msgr2=0x7fcb0c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 shutdown_connections 2024-03-28T18:38:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.144+0000 7fcb1224b640 1 -- 172.21.15.145:0/580357814 wait complete. 2024-03-28T18:38:02.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:38:02.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:01 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3114279970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 -- 172.21.15.145:0/2026180917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e0101790 msgr2=0x7ff9e0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/2026180917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e0101790 0x7ff9e0101c10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009860 tx=0x7ff9d402bb70 comp rx=0 tx=0).stop 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 -- 172.21.15.145:0/2026180917 shutdown_connections 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/2026180917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/2026180917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e0101790 0x7ff9e0101c10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/2026180917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e01005a0 0x7ff9e01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.263 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 -- 172.21.15.145:0/2026180917 >> 172.21.15.145:0/2026180917 conn(0x7ff9e00fbb30 msgr2=0x7ff9e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.264 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 -- 172.21.15.145:0/2026180917 shutdown_connections 2024-03-28T18:38:02.264 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.263+0000 7ff9e67f4640 1 -- 172.21.15.145:0/2026180917 wait complete. 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 Processor -- start 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 -- start start 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e01005a0 0x7ff9e0104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010c230 con 0x7ff9e0102150 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010c0b0 con 0x7ff9e01005a0 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e67f4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010c3b0 con 0x7ff9e0101790 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9df7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e4d91640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9dffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e01005a0 0x7ff9e0104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.266 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9df7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:55128/0 (socket says 172.21.15.145:55128) 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9df7fe640 1 -- 172.21.15.145:0/223343095 learned_addr learned my addr 172.21.15.145:0/223343095 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.264+0000 7ff9e4d91640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54024/0 (socket says 172.21.15.145:54024) 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 -- 172.21.15.145:0/223343095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e01005a0 msgr2=0x7ff9e0104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e01005a0 0x7ff9e0104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 -- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 msgr2=0x7ff9e01030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 -- 172.21.15.145:0/223343095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0103940 con 0x7ff9e0101790 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9e4d91640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9df7fe640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40349e0 tx=0x7ff9d4032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4041030 con 0x7ff9e0101790 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d402c510 con 0x7ff9e0101790 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e01a1a80 con 0x7ff9e0101790 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4039040 con 0x7ff9e0101790 2024-03-28T18:38:02.267 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.265+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e01a1fd0 con 0x7ff9e0101790 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.266+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e01a21a0 con 0x7ff9e0101790 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.267+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff9d4044070 con 0x7ff9e0101790 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.268+0000 7ff9dd7fa640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 0x7ff9c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.268+0000 7ff9dffff640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 0x7ff9c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.268+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff9d40f0350 con 0x7ff9e0101790 2024-03-28T18:38:02.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.268+0000 7ff9dffff640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 0x7ff9c80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff9e01015f0 tx=0x7ff9d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.272+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff9d40bd040 con 0x7ff9e0101790 2024-03-28T18:38:02.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.453+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff9e01a2370 con 0x7ff9e0101790 2024-03-28T18:38:02.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.454+0000 7ff9dd7fa640 1 -- 172.21.15.145:0/223343095 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff9d40bc940 con 0x7ff9e0101790 2024-03-28T18:38:02.454 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:38:02.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 msgr2=0x7ff9c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 0x7ff9c80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff9e01015f0 tx=0x7ff9d0005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:02.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 msgr2=0x7ff9e0102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40349e0 tx=0x7ff9d4032450 comp rx=0 tx=0).stop 2024-03-28T18:38:02.456 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 shutdown_connections 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e0102150 0x7ff9e01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9c80a2800 0x7ff9c80a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e0101790 0x7ff9e0102b70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 --2- 172.21.15.145:0/223343095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e01005a0 0x7ff9e0104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 >> 172.21.15.145:0/223343095 conn(0x7ff9e00fbb30 msgr2=0x7ff9e00fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 shutdown_connections 2024-03-28T18:38:02.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.456+0000 7ff9e67f4640 1 -- 172.21.15.145:0/223343095 wait complete. 2024-03-28T18:38:02.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 -- 172.21.15.145:0/2602242727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 msgr2=0x7ff168106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 --2- 172.21.15.145:0/2602242727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168106e40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff154009860 tx=0x7ff15402bb70 comp rx=0 tx=0).stop 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 -- 172.21.15.145:0/2602242727 shutdown_connections 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 --2- 172.21.15.145:0/2602242727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff1681097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 --2- 172.21.15.145:0/2602242727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168106e40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 --2- 172.21.15.145:0/2602242727 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 -- 172.21.15.145:0/2602242727 >> 172.21.15.145:0/2602242727 conn(0x7ff1680fbb30 msgr2=0x7ff1680fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.582 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 -- 172.21.15.145:0/2602242727 shutdown_connections 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.580+0000 7ff16eb80640 1 -- 172.21.15.145:0/2602242727 wait complete. 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 Processor -- start 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 -- start start 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff16819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16810d300 con 0x7ff168104a50 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16810d180 con 0x7ff168102120 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16eb80640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16810d480 con 0x7ff168107380 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff167fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16c91c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16d11d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff16819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16c91c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:59856/0 (socket says 172.21.15.145:59856) 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff16c91c640 1 -- 172.21.15.145:0/3406000077 learned_addr learned my addr 172.21.15.145:0/3406000077 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.581+0000 7ff167fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56502/0 (socket says 172.21.15.145:56502) 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 -- 172.21.15.145:0/3406000077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 msgr2=0x7ff16819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff16819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 -- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 msgr2=0x7ff168197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 -- 172.21.15.145:0/3406000077 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16819f650 con 0x7ff168102120 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16d11d640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff16819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff167fff640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16c91c640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00ea60 tx=0x7ff15c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c00cde0 con 0x7ff168102120 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff15c0035f0 con 0x7ff168102120 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16819f940 con 0x7ff168102120 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c010670 con 0x7ff168102120 2024-03-28T18:38:02.584 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.582+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16819feb0 con 0x7ff168102120 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.583+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1680fd620 con 0x7ff168102120 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.584+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff15c0037a0 con 0x7ff168102120 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.585+0000 7ff165ffb640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 0x7ff1580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.585+0000 7ff167fff640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 0x7ff1580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.585+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff15c0cf0b0 con 0x7ff168102120 2024-03-28T18:38:02.585 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.585+0000 7ff167fff640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 0x7ff1580a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff154009830 tx=0x7ff15402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.588 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.588+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff15c09bd40 con 0x7ff168102120 2024-03-28T18:38:02.772 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.773+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff1680622a0 con 0x7ff168102120 2024-03-28T18:38:02.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.773+0000 7ff165ffb640 1 -- 172.21.15.145:0/3406000077 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff15c097030 con 0x7ff168102120 2024-03-28T18:38:02.773 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.775+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 msgr2=0x7ff1580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.775+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 0x7ff1580a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff154009830 tx=0x7ff15402e040 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.775+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 msgr2=0x7ff168197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.775+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00ea60 tx=0x7ff15c00ef20 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 shutdown_connections 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff1580a2880 0x7ff1580a4d40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff168107380 0x7ff16819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff168104a50 0x7ff168197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 --2- 172.21.15.145:0/3406000077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff168102120 0x7ff168197a30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 >> 172.21.15.145:0/3406000077 conn(0x7ff1680fbb30 msgr2=0x7ff168102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 shutdown_connections 2024-03-28T18:38:02.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.776+0000 7ff16eb80640 1 -- 172.21.15.145:0/3406000077 wait complete. 2024-03-28T18:38:02.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:02 smithi145 ceph-mon[32151]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:38:02.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:02 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1722272630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:02.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:02 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/580357814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:02.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:02 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/223343095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:02.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:38:02.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 -- 172.21.15.145:0/663565468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0104b10 msgr2=0x7f49a0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 --2- 172.21.15.145:0/663565468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0104b10 0x7f49a0106f00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4994009900 tx=0x7f499402bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:02.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 -- 172.21.15.145:0/663565468 shutdown_connections 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 --2- 172.21.15.145:0/663565468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a0107440 0x7f49a0109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 --2- 172.21.15.145:0/663565468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0104b10 0x7f49a0106f00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 --2- 172.21.15.145:0/663565468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a01021e0 0x7f49a01045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 -- 172.21.15.145:0/663565468 >> 172.21.15.145:0/663565468 conn(0x7f49a00fbb30 msgr2=0x7f49a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 -- 172.21.15.145:0/663565468 shutdown_connections 2024-03-28T18:38:02.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.891+0000 7f49a54c6640 1 -- 172.21.15.145:0/663565468 wait complete. 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 Processor -- start 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 -- start start 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 0x7f49a01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a0104b10 0x7f49a0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 0x7f49a019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a010d310 con 0x7f49a0107440 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a010d190 con 0x7f49a01021e0 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f49a54c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a010d490 con 0x7f49a0104b10 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f499f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 0x7f49a019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.892+0000 7f499effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 0x7f49a01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 0x7f49a019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56518/0 (socket says 172.21.15.145:56518) 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499f7fe640 1 -- 172.21.15.145:0/907889139 learned_addr learned my addr 172.21.15.145:0/907889139 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499e7fc640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a0104b10 0x7f49a0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 -- 172.21.15.145:0/907889139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a0104b10 msgr2=0x7f49a0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a0104b10 0x7f49a0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 -- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 msgr2=0x7f49a019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 0x7f49a019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 -- 172.21.15.145:0/907889139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a019f600 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f499effd640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 0x7f49a01979e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9f0 tx=0x7f499000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499000ccc0 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4990010910 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a019f8f0 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990018960 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.893+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a019fe10 con 0x7f49a01021e0 2024-03-28T18:38:02.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.894+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a00fd620 con 0x7f49a01021e0 2024-03-28T18:38:02.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.896+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4990010430 con 0x7f49a01021e0 2024-03-28T18:38:02.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.896+0000 7f4987fff640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 0x7f49880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:02.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.896+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f49900cf1f0 con 0x7f49a01021e0 2024-03-28T18:38:02.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.897+0000 7f499e7fc640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 0x7f49880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:02.897 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.897+0000 7f499e7fc640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 0x7f49880a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4994000c00 tx=0x7f4994005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:02.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:02.905+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f499009b890 con 0x7f49a01021e0 2024-03-28T18:38:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:02 smithi164 ceph-mon[36248]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:38:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:02 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1722272630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:02 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/580357814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:02 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/223343095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.057 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:02 smithi192 ceph-mon[36589]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:38:03.058 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:02 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1722272630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.058 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:02 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/580357814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.058 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:02 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/223343095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.130+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a002d070 con 0x7f49a01021e0 2024-03-28T18:38:03.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.131+0000 7f4987fff640 1 -- 172.21.15.145:0/907889139 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7f49a002d070 con 0x7f49a01021e0 2024-03-28T18:38:03.131 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:38:03.131 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:38:03.131 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 msgr2=0x7f49880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 0x7f49880a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4994000c00 tx=0x7f4994005fd0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 msgr2=0x7f49a01979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 0x7f49a01979e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9f0 tx=0x7f499000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 shutdown_connections 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49a0107440 0x7f49a019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f49880a2800 0x7f49880a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49a0104b10 0x7f49a0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 --2- 172.21.15.145:0/907889139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49a01021e0 0x7f49a01979e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.133 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 >> 172.21.15.145:0/907889139 conn(0x7f49a00fbb30 msgr2=0x7f49a0102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 shutdown_connections 2024-03-28T18:38:03.134 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.133+0000 7f49a54c6640 1 -- 172.21.15.145:0/907889139 wait complete. 2024-03-28T18:38:03.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- 172.21.15.145:0/1225037195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07301016e0 msgr2=0x7f0730101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 --2- 172.21.15.145:0/1225037195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07301016e0 0x7f0730101b00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f072c009900 tx=0x7f072c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- 172.21.15.145:0/1225037195 shutdown_connections 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 --2- 172.21.15.145:0/1225037195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0730103290 0x7f0730109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 --2- 172.21.15.145:0/1225037195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301028d0 0x7f0730102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 --2- 172.21.15.145:0/1225037195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07301016e0 0x7f0730101b00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.262 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- 172.21.15.145:0/1225037195 >> 172.21.15.145:0/1225037195 conn(0x7f07300fcc50 msgr2=0x7f07300ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.265 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- 172.21.15.145:0/1225037195 shutdown_connections 2024-03-28T18:38:03.265 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- 172.21.15.145:0/1225037195 wait complete. 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 Processor -- start 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.262+0000 7f0738532640 1 -- start start 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301016e0 0x7f0730197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07301028d0 0x7f0730197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073010d350 con 0x7f0730103290 2024-03-28T18:38:03.268 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073010d1d0 con 0x7f07301028d0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073010d4d0 con 0x7f07301016e0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56536/0 (socket says 172.21.15.145:56536) 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 -- 172.21.15.145:0/1933526854 learned_addr learned my addr 172.21.15.145:0/1933526854 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0735acd640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07301028d0 0x7f0730197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f07362ce640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301016e0 0x7f0730197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 -- 172.21.15.145:0/1933526854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301016e0 msgr2=0x7f0730197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301016e0 0x7f0730197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 -- 172.21.15.145:0/1933526854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07301028d0 msgr2=0x7f0730197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07301028d0 0x7f0730197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 -- 172.21.15.145:0/1933526854 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073019f670 con 0x7f0730103290 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0736acf640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f072800eb10 tx=0x7f072800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728004b20 con 0x7f0730103290 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0728004cc0 con 0x7f0730103290 2024-03-28T18:38:03.269 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073019f960 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728010760 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.263+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073019feb0 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.265+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0730108ad0 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.266+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0728005000 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.267+0000 7f07277fe640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 0x7f071c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.267+0000 7f07362ce640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 0x7f071c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.267+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f07280cf370 con 0x7f0730103290 2024-03-28T18:38:03.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.268+0000 7f07362ce640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 0x7f071c0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f072c000c00 tx=0x7f072c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.272+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f072809c060 con 0x7f0730103290 2024-03-28T18:38:03.473 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.474+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07300622a0 con 0x7f0730103290 2024-03-28T18:38:03.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.476+0000 7f07277fe640 1 -- 172.21.15.145:0/1933526854 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7f072809b960 con 0x7f0730103290 2024-03-28T18:38:03.476 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:38:03.477 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:38:03.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 msgr2=0x7f071c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 0x7f071c0a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f072c000c00 tx=0x7f072c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 msgr2=0x7f073019f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f072800eb10 tx=0x7f072800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 shutdown_connections 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0730103290 0x7f073019f000 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07301028d0 0x7f0730197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f071c0a2880 0x7f071c0a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 --2- 172.21.15.145:0/1933526854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07301016e0 0x7f0730197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 >> 172.21.15.145:0/1933526854 conn(0x7f07300fcc50 msgr2=0x7f0730105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 shutdown_connections 2024-03-28T18:38:03.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.479+0000 7f0738532640 1 -- 172.21.15.145:0/1933526854 wait complete. 2024-03-28T18:38:03.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:38:03.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- 172.21.15.145:0/3862901056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 msgr2=0x7fea3c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3862901056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c1089e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea34009ca0 tx=0x7fea3402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- 172.21.15.145:0/3862901056 shutdown_connections 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3862901056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c1089e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3862901056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c101790 0x7fea3c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3862901056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c1005a0 0x7fea3c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- 172.21.15.145:0/3862901056 >> 172.21.15.145:0/3862901056 conn(0x7fea3c0fbb30 msgr2=0x7fea3c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- 172.21.15.145:0/3862901056 shutdown_connections 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- 172.21.15.145:0/3862901056 wait complete. 2024-03-28T18:38:03.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 Processor -- start 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.601+0000 7fea40aa1640 1 -- start start 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 0x7fea3c197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c10d300 con 0x7fea3c101790 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c10d180 con 0x7fea3c1005a0 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c10d480 con 0x7fea3c102150 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56542/0 (socket says 172.21.15.145:56542) 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea3a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 0x7fea3c197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 -- 172.21.15.145:0/3205725275 learned_addr learned my addr 172.21.15.145:0/3205725275 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea3ad9d640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 -- 172.21.15.145:0/3205725275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 msgr2=0x7fea3c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 -- 172.21.15.145:0/3205725275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 msgr2=0x7fea3c197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 0x7fea3c197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 -- 172.21.15.145:0/3205725275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c19f560 con 0x7fea3c101790 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea3a59c640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 0x7fea3c197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea3ad9d640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea39d9b640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fea3000ea60 tx=0x7fea3000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000cde0 con 0x7fea3c101790 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c19f850 con 0x7fea3c101790 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea300035f0 con 0x7fea3c101790 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30010670 con 0x7fea3c101790 2024-03-28T18:38:03.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.602+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c19fda0 con 0x7fea3c101790 2024-03-28T18:38:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.603+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c1078c0 con 0x7fea3c101790 2024-03-28T18:38:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.605+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fea300037a0 con 0x7fea3c101790 2024-03-28T18:38:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.606+0000 7fea2b7fe640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 0x7fea240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.606+0000 7fea3a59c640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 0x7fea240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.606+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fea300cf1c0 con 0x7fea3c101790 2024-03-28T18:38:03.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.606+0000 7fea3a59c640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 0x7fea240a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fea3c1015f0 tx=0x7fea2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.612+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fea3009bf40 con 0x7fea3c101790 2024-03-28T18:38:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:03 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3406000077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:03 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/907889139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:03 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1933526854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:03.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.861+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea3c02d200 con 0x7fea3c101790 2024-03-28T18:38:03.861 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.862+0000 7fea2b7fe640 1 -- 172.21.15.145:0/3205725275 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea3009b840 con 0x7fea3c101790 2024-03-28T18:38:03.862 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 msgr2=0x7fea240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 0x7fea240a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fea3c1015f0 tx=0x7fea2c005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 msgr2=0x7fea3c197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fea3000ea60 tx=0x7fea3000ef20 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 shutdown_connections 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fea240a2800 0x7fea240a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea3c102150 0x7fea3c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea3c101790 0x7fea3c197e80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 --2- 172.21.15.145:0/3205725275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3c1005a0 0x7fea3c197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 >> 172.21.15.145:0/3205725275 conn(0x7fea3c0fbb30 msgr2=0x7fea3c104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 shutdown_connections 2024-03-28T18:38:03.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.864+0000 7fea40aa1640 1 -- 172.21.15.145:0/3205725275 wait complete. 2024-03-28T18:38:03.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:38:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:03 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3406000077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:03 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/907889139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:03.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:03 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1933526854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 -- 172.21.15.145:0/3178448459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2498102150 msgr2=0x7f24981089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 --2- 172.21.15.145:0/3178448459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2498102150 0x7f24981089e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2490009ca0 tx=0x7f249002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 -- 172.21.15.145:0/3178448459 shutdown_connections 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 --2- 172.21.15.145:0/3178448459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2498102150 0x7f24981089e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 --2- 172.21.15.145:0/3178448459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498101790 0x7f2498101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 --2- 172.21.15.145:0/3178448459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24981005a0 0x7f24981009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 -- 172.21.15.145:0/3178448459 >> 172.21.15.145:0/3178448459 conn(0x7f24980fbb30 msgr2=0x7f24980fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 -- 172.21.15.145:0/3178448459 shutdown_connections 2024-03-28T18:38:03.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 -- 172.21.15.145:0/3178448459 wait complete. 2024-03-28T18:38:03.982 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.980+0000 7f24a00b2640 1 Processor -- start 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 -- start start 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498101790 0x7f249819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498102150 0x7f249819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249810c230 con 0x7f24981005a0 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249810c0b0 con 0x7f2498102150 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f24a00b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249810c3b0 con 0x7f2498101790 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56564/0 (socket says 172.21.15.145:56564) 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249d64d640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498101790 0x7f249819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 -- 172.21.15.145:0/4028116667 learned_addr learned my addr 172.21.15.145:0/4028116667 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249e64f640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498102150 0x7f249819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 -- 172.21.15.145:0/4028116667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498101790 msgr2=0x7f249819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498101790 0x7f249819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.983 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 -- 172.21.15.145:0/4028116667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498102150 msgr2=0x7f249819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498102150 0x7f249819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 -- 172.21.15.145:0/4028116667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249819f590 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f249de4e640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9c0 tx=0x7f249400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.981+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249400cd80 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.982+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24940035f0 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.982+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494010690 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.982+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249819f880 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.982+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249819fdd0 con 0x7f24981005a0 2024-03-28T18:38:03.984 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.983+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24981078c0 con 0x7f24981005a0 2024-03-28T18:38:03.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.984+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f24940037a0 con 0x7f24981005a0 2024-03-28T18:38:03.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.984+0000 7f248effd640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 0x7f24840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:03.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.984+0000 7f249d64d640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 0x7f24840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:03.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.984+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f24940cf0e0 con 0x7f24981005a0 2024-03-28T18:38:03.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.985+0000 7f249d64d640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 0x7f24840a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2498069100 tx=0x7f2488005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:03.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:03.988+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f249409bdd0 con 0x7f24981005a0 2024-03-28T18:38:04.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:03 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3406000077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:03 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/907889139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:04.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:03 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1933526854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:04.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.169+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24980622d0 con 0x7f24981005a0 2024-03-28T18:38:04.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.169+0000 7f248effd640 1 -- 172.21.15.145:0/4028116667 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f249409b6d0 con 0x7f24981005a0 2024-03-28T18:38:04.169 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:38:04.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 msgr2=0x7f24840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 0x7f24840a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2498069100 tx=0x7f2488005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 msgr2=0x7f2498069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9c0 tx=0x7f249400ee80 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 shutdown_connections 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2498102150 0x7f249819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f24840a2880 0x7f24840a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498101790 0x7f249819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 --2- 172.21.15.145:0/4028116667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24981005a0 0x7f2498069390 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 >> 172.21.15.145:0/4028116667 conn(0x7f24980fbb30 msgr2=0x7f2498104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 shutdown_connections 2024-03-28T18:38:04.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.171+0000 7f24a00b2640 1 -- 172.21.15.145:0/4028116667 wait complete. 2024-03-28T18:38:04.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:38:04.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- 172.21.15.145:0/2710014709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e81005a0 msgr2=0x7f95e81009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/2710014709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e81005a0 0x7f95e81009c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009900 tx=0x7f95d802bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:04.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- 172.21.15.145:0/2710014709 shutdown_connections 2024-03-28T18:38:04.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/2710014709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 0x7f95e81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.292 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/2710014709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e8101790 0x7f95e8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/2710014709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e81005a0 0x7f95e81009c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- 172.21.15.145:0/2710014709 >> 172.21.15.145:0/2710014709 conn(0x7f95e80fbb30 msgr2=0x7f95e80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- 172.21.15.145:0/2710014709 shutdown_connections 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- 172.21.15.145:0/2710014709 wait complete. 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 Processor -- start 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- start start 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e81005a0 0x7f95e8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 0x7f95e819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e810c230 con 0x7f95e8101790 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e810c0b0 con 0x7f95e81005a0 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.288+0000 7f95ee2b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e810c3b0 con 0x7f95e8102150 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56586/0 (socket says 172.21.15.145:56586) 2024-03-28T18:38:04.293 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 -- 172.21.15.145:0/3798361075 learned_addr learned my addr 172.21.15.145:0/3798361075 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e7fff640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e81005a0 0x7f95e8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95ec84f640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 0x7f95e819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 -- 172.21.15.145:0/3798361075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 msgr2=0x7f95e819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 0x7f95e819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 -- 172.21.15.145:0/3798361075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e81005a0 msgr2=0x7f95e8197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e81005a0 0x7f95e8197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 -- 172.21.15.145:0/3798361075 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e819f630 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e77fe640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00e9f0 tx=0x7f95dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00cd80 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95dc010910 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e819f920 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc018990 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.289+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e819fe70 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.290+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e81a00a0 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.291+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f95dc010ac0 con 0x7f95e8101790 2024-03-28T18:38:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.292+0000 7f95e57fa640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.295 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.292+0000 7f95e7fff640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.295 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.292+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f95dc0cf530 con 0x7f95e8101790 2024-03-28T18:38:04.295 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.293+0000 7f95e7fff640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 0x7f95d00a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f95d8000c00 tx=0x7f95d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:04.295 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.295+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95dc09bbd0 con 0x7f95e8101790 2024-03-28T18:38:04.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.477+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95e81a0270 con 0x7f95e8101790 2024-03-28T18:38:04.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.478+0000 7f95e57fa640 1 -- 172.21.15.145:0/3798361075 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95dc09fc70 con 0x7f95e8101790 2024-03-28T18:38:04.478 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 msgr2=0x7f95d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 0x7f95d00a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f95d8000c00 tx=0x7f95d8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 msgr2=0x7f95e8197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00e9f0 tx=0x7f95dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 shutdown_connections 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f95d00a2800 0x7f95d00a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95e8102150 0x7f95e819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95e8101790 0x7f95e8197f50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 --2- 172.21.15.145:0/3798361075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95e81005a0 0x7f95e8197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 >> 172.21.15.145:0/3798361075 conn(0x7f95e80fbb30 msgr2=0x7f95e80fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 shutdown_connections 2024-03-28T18:38:04.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.480+0000 7f95ee2b2640 1 -- 172.21.15.145:0/3798361075 wait complete. 2024-03-28T18:38:04.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:38:04.600 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 -- 172.21.15.145:0/517031921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 msgr2=0x7fcf58109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 --2- 172.21.15.145:0/517031921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf58109810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009ca0 tx=0x7fcf5002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 -- 172.21.15.145:0/517031921 shutdown_connections 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 --2- 172.21.15.145:0/517031921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf58109810 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 --2- 172.21.15.145:0/517031921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf58104af0 0x7fcf58106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 --2- 172.21.15.145:0/517031921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf581021c0 0x7fcf581045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 -- 172.21.15.145:0/517031921 >> 172.21.15.145:0/517031921 conn(0x7fcf580fbb30 msgr2=0x7fcf580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.600+0000 7fcf5d434640 1 -- 172.21.15.145:0/517031921 shutdown_connections 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 -- 172.21.15.145:0/517031921 wait complete. 2024-03-28T18:38:04.601 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 Processor -- start 2024-03-28T18:38:04.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 -- start start 2024-03-28T18:38:04.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 0x7fcf581979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf58104af0 0x7fcf58197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5810d340 con 0x7fcf58107420 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5810d1c0 con 0x7fcf581021c0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.601+0000 7fcf5d434640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5810d4c0 con 0x7fcf58104af0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 0x7fcf581979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56596/0 (socket says 172.21.15.145:56596) 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 -- 172.21.15.145:0/1543375096 learned_addr learned my addr 172.21.15.145:0/1543375096 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 0x7fcf581979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:59942/0 (socket says 172.21.15.145:59942) 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf567fc640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf58104af0 0x7fcf58197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 -- 172.21.15.145:0/1543375096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf58104af0 msgr2=0x7fcf58197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf58104af0 0x7fcf58197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 -- 172.21.15.145:0/1543375096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 msgr2=0x7fcf581979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 0x7fcf581979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 -- 172.21.15.145:0/1543375096 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5819f610 con 0x7fcf58107420 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf577fe640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf5002c630 tx=0x7fcf50032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50041020 con 0x7fcf58107420 2024-03-28T18:38:04.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf50034cd0 con 0x7fcf58107420 2024-03-28T18:38:04.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5819f8a0 con 0x7fcf58107420 2024-03-28T18:38:04.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50039040 con 0x7fcf58107420 2024-03-28T18:38:04.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.602+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5819fde0 con 0x7fcf58107420 2024-03-28T18:38:04.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.604+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf580fd620 con 0x7fcf58107420 2024-03-28T18:38:04.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.604+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcf50044020 con 0x7fcf58107420 2024-03-28T18:38:04.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.605+0000 7fcf4ffff640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:04.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.605+0000 7fcf56ffd640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:04.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.605+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fcf500f0ba0 con 0x7fcf58107420 2024-03-28T18:38:04.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.606+0000 7fcf56ffd640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 0x7fcf440a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009bd0 tx=0x7fcf48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:04.609 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.609+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf5004b020 con 0x7fcf58107420 2024-03-28T18:38:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:04 smithi145 ceph-mon[32151]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-28T18:38:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3205725275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4028116667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:04 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3798361075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.891+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcf580622a0 con 0x7fcf58107420 2024-03-28T18:38:04.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.891+0000 7fcf4ffff640 1 -- 172.21.15.145:0/1543375096 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcf500bd920 con 0x7fcf58107420 2024-03-28T18:38:04.891 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 msgr2=0x7fcf440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 0x7fcf440a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009bd0 tx=0x7fcf48005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 msgr2=0x7fcf5819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcf5002c630 tx=0x7fcf50032ea0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 shutdown_connections 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf58107420 0x7fcf5819efa0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fcf440a2880 0x7fcf440a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf58104af0 0x7fcf58197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.894 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 --2- 172.21.15.145:0/1543375096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf581021c0 0x7fcf581979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 >> 172.21.15.145:0/1543375096 conn(0x7fcf580fbb30 msgr2=0x7fcf58102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 shutdown_connections 2024-03-28T18:38:04.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:04.894+0000 7fcf5d434640 1 -- 172.21.15.145:0/1543375096 wait complete. 2024-03-28T18:38:04.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:38:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:04 smithi164 ceph-mon[36248]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-28T18:38:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3205725275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4028116667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:04 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3798361075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 -- 172.21.15.145:0/896800894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 msgr2=0x7f9d44106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 --2- 172.21.15.145:0/896800894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44106e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d3802bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 -- 172.21.15.145:0/896800894 shutdown_connections 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 --2- 172.21.15.145:0/896800894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d441097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 --2- 172.21.15.145:0/896800894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44106e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 --2- 172.21.15.145:0/896800894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 -- 172.21.15.145:0/896800894 >> 172.21.15.145:0/896800894 conn(0x7f9d440fbb30 msgr2=0x7f9d440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 -- 172.21.15.145:0/896800894 shutdown_connections 2024-03-28T18:38:05.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.008+0000 7f9d49df3640 1 -- 172.21.15.145:0/896800894 wait complete. 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 Processor -- start 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 -- start start 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d4419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4410d340 con 0x7f9d44104a70 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4410d1c0 con 0x7f9d44102140 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d49df3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4410d4c0 con 0x7f9d441073a0 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d42ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56604/0 (socket says 172.21.15.145:56604) 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d42ffd640 1 -- 172.21.15.145:0/405547726 learned_addr learned my addr 172.21.15.145:0/405547726 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d43fff640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d4419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.009+0000 7f9d437fe640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 -- 172.21.15.145:0/405547726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 msgr2=0x7f9d4419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d4419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 -- 172.21.15.145:0/405547726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 msgr2=0x7f9d44197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 -- 172.21.15.145:0/405547726 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4419f630 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d43fff640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d4419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d437fe640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d42ffd640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9d38000c00 tx=0x7f9d3802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38041070 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4419f8c0 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d38004450 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4419fe00 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.010+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38034440 con 0x7f9d44104a70 2024-03-28T18:38:05.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.011+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d440fd620 con 0x7f9d44104a70 2024-03-28T18:38:05.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.012+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9d38044070 con 0x7f9d44104a70 2024-03-28T18:38:05.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.013+0000 7f9d40ff9640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 0x7f9d2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.013+0000 7f9d437fe640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 0x7f9d2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.013+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f9d380f0ab0 con 0x7f9d44104a70 2024-03-28T18:38:05.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.013+0000 7f9d437fe640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 0x7f9d2c0a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009fc0 tx=0x7f9d34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.016+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d380bd1e0 con 0x7f9d44104a70 2024-03-28T18:38:05.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:04 smithi192 ceph-mon[36589]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-28T18:38:05.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3205725275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4028116667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:04 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3798361075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.258 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.258+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d4402d070 con 0x7f9d44104a70 2024-03-28T18:38:05.258 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.258+0000 7f9d40ff9640 1 -- 172.21.15.145:0/405547726 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11481 (secure 0 0 0) 0x7f9d380c1280 con 0x7f9d44104a70 2024-03-28T18:38:05.259 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:38:05.259 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:36:13.026843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24457":{"gid":24457,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/1226474966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1226474966},{"type":"v1","addr":"172.21.15.192:6835","nonce":1226474966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24469,24491,14631,24461,24496]},"id":1}]} 2024-03-28T18:38:05.259 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 28 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 msgr2=0x7f9d2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 0x7f9d2c0a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009fc0 tx=0x7f9d34005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 msgr2=0x7f9d44197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9d38000c00 tx=0x7f9d3802c640 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 shutdown_connections 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d2c0a2800 0x7f9d2c0a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d441073a0 0x7f9d4419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d44104a70 0x7f9d44197f50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 --2- 172.21.15.145:0/405547726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d44102140 0x7f9d44197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 >> 172.21.15.145:0/405547726 conn(0x7f9d440fbb30 msgr2=0x7f9d44102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 shutdown_connections 2024-03-28T18:38:05.261 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.261+0000 7f9d49df3640 1 -- 172.21.15.145:0/405547726 wait complete. 2024-03-28T18:38:05.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 -- 172.21.15.145:0/2887960594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 msgr2=0x7f2160108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 --2- 172.21.15.145:0/2887960594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f2160108a00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2158009ca0 tx=0x7f215802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 -- 172.21.15.145:0/2887960594 shutdown_connections 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 --2- 172.21.15.145:0/2887960594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f2160108a00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 --2- 172.21.15.145:0/2887960594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 0x7f2160101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 --2- 172.21.15.145:0/2887960594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 0x7f21601009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.377 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 -- 172.21.15.145:0/2887960594 >> 172.21.15.145:0/2887960594 conn(0x7f21600fbb30 msgr2=0x7f21600fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 -- 172.21.15.145:0/2887960594 shutdown_connections 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.375+0000 7f21684c0640 1 -- 172.21.15.145:0/2887960594 wait complete. 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 Processor -- start 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 -- start start 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 0x7f21601968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 0x7f2160196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010c240 con 0x7f2160102170 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010c0c0 con 0x7f21601017b0 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f21684c0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010c3c0 con 0x7f21601005c0 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f2166a5d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f2166a5d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56620/0 (socket says 172.21.15.145:56620) 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f2166a5d640 1 -- 172.21.15.145:0/1286580902 learned_addr learned my addr 172.21.15.145:0/1286580902 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f216625c640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 0x7f21601968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.376+0000 7f2165a5b640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 0x7f2160196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 -- 172.21.15.145:0/1286580902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 msgr2=0x7f21601968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 0x7f21601968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 -- 172.21.15.145:0/1286580902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 msgr2=0x7f2160196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 0x7f2160196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 -- 172.21.15.145:0/1286580902 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216019e4d0 con 0x7f2160102170 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f2166a5d640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2158037040 tx=0x7f2158032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158034780 con 0x7f2160102170 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2158032690 con 0x7f2160102170 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158004040 con 0x7f2160102170 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2160068800 con 0x7f2160102170 2024-03-28T18:38:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.377+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2160068cf0 con 0x7f2160102170 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.378+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2160068490 con 0x7f2160102170 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.379+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2158034920 con 0x7f2160102170 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.380+0000 7f21577fe640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 0x7f214c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.380+0000 7f216625c640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 0x7f214c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.380+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f21580f2a70 con 0x7f2160102170 2024-03-28T18:38:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.380+0000 7f216625c640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 0x7f214c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2160101610 tx=0x7f215c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.385 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.386+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21580bf7f0 con 0x7f2160102170 2024-03-28T18:38:05.568 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.568+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f216019e6a0 con 0x7f2160102170 2024-03-28T18:38:05.570 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.571+0000 7f21577fe640 1 -- 172.21.15.145:0/1286580902 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7f21580bf0f0 con 0x7f2160102170 2024-03-28T18:38:05.571 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:38:05.571 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:38:05.573 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 msgr2=0x7f214c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 0x7f214c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2160101610 tx=0x7f215c005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 msgr2=0x7f216019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2158037040 tx=0x7f2158032f20 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 shutdown_connections 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.573+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2160102170 0x7f216019de60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21601017b0 0x7f2160196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f214c0a2800 0x7f214c0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 --2- 172.21.15.145:0/1286580902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21601005c0 0x7f21601968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 >> 172.21.15.145:0/1286580902 conn(0x7f21600fbb30 msgr2=0x7f2160104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 shutdown_connections 2024-03-28T18:38:05.574 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.574+0000 7f21684c0640 1 -- 172.21.15.145:0/1286580902 wait complete. 2024-03-28T18:38:05.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:35:50.266038+0000', 'modified': '2024-03-28T18:36:13.026843+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24496, 'mds_1': 24491, 'mds_2': 24469, 'mds_3': 24461, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.145:6839/1703430127', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6838', 'nonce': 1703430127}, {'type': 'v1', 'addr': '172.21.15.145:6839', 'nonce': 1703430127}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.145:6841/2292179731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6840', 'nonce': 2292179731}, {'type': 'v1', 'addr': '172.21.15.145:6841', 'nonce': 2292179731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6833/3835667312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6832', 'nonce': 3835667312}, {'type': 'v1', 'addr': '172.21.15.164:6833', 'nonce': 3835667312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6833/3704007459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 3704007459}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 3704007459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6835/3793155920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 3793155920}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 3793155920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/1226474966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 1226474966}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 1226474966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.164:6837/1634173775', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 1634173775}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 1634173775}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.192:6837/2153191464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 2153191464}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 2153191464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24491': {'gid': 24491, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.164:6839/644460294', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6838', 'nonce': 644460294}, {'type': 'v1', 'addr': '172.21.15.164:6839', 'nonce': 644460294}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.192:6839/2516430938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 2516430938}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 2516430938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24496, 'qdb_cluster': [24469, 24491, 14631, 24461, 24496]} 2024-03-28T18:38:05.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T18:38:05.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 -- 172.21.15.145:0/2333799064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce41005b0 msgr2=0x7f3ce41009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 --2- 172.21.15.145:0/2333799064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce41005b0 0x7f3ce41009d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd802bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 -- 172.21.15.145:0/2333799064 shutdown_connections 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 --2- 172.21.15.145:0/2333799064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce4102160 0x7f3ce41089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 --2- 172.21.15.145:0/2333799064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 0x7f3ce4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 --2- 172.21.15.145:0/2333799064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce41005b0 0x7f3ce41009d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 -- 172.21.15.145:0/2333799064 >> 172.21.15.145:0/2333799064 conn(0x7f3ce40fbb40 msgr2=0x7f3ce40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 -- 172.21.15.145:0/2333799064 shutdown_connections 2024-03-28T18:38:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.694+0000 7f3cea86c640 1 -- 172.21.15.145:0/2333799064 wait complete. 2024-03-28T18:38:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 Processor -- start 2024-03-28T18:38:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 -- start start 2024-03-28T18:38:05.696 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce41005b0 0x7f3ce406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 0x7f3ce419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce410c200 con 0x7f3ce4102160 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce410c080 con 0x7f3ce41017a0 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3cea86c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce410c380 con 0x7f3ce41005b0 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56628/0 (socket says 172.21.15.145:56628) 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 -- 172.21.15.145:0/388643707 learned_addr learned my addr 172.21.15.145:0/388643707 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce41005b0 0x7f3ce406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 0x7f3ce419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 -- 172.21.15.145:0/388643707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce41005b0 msgr2=0x7f3ce406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce41005b0 0x7f3ce406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 -- 172.21.15.145:0/388643707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 msgr2=0x7f3ce419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 0x7f3ce419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.697 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 -- 172.21.15.145:0/388643707 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce41a1860 con 0x7f3ce4102160 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.695+0000 7f3ce8e09640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00eb10 tx=0x7f3cdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.696+0000 7f3ce17fa640 1 -- 172.21.15.145:0/388643707 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc004b20 con 0x7f3ce4102160 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.696+0000 7f3ce17fa640 1 -- 172.21.15.145:0/388643707 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc004cc0 con 0x7f3ce4102160 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.696+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce41a1af0 con 0x7f3ce4102160 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.696+0000 7f3ce17fa640 1 -- 172.21.15.145:0/388643707 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc010760 con 0x7f3ce4102160 2024-03-28T18:38:05.698 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.696+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce41a1ff0 con 0x7f3ce4102160 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ce17fa640 1 -- 172.21.15.145:0/388643707 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3cdc005020 con 0x7f3ce4102160 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 Processor -- start 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 -- start start 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89b740 0x55c1aa89bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8a3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa800c20 con 0x55c1aa8a12a0 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa7a9a40 con 0x55c1aa89b740 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ced458740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa7fccc0 con 0x55c1aa89c130 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ce17fa640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 0x7f3ccc0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.698+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:48072/0 (socket says 172.21.15.145:48072) 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 -- 172.21.15.145:0/3469344420 learned_addr learned my addr 172.21.15.145:0/3469344420 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89b740 0x55c1aa89bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce8e09640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 0x7f3ccc0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 -- 172.21.15.145:0/3469344420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89b740 msgr2=0x55c1aa89bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89b740 0x55c1aa89bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 -- 172.21.15.145:0/3469344420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 msgr2=0x55c1aa8a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 -- 172.21.15.145:0/3469344420 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1aa7cbf90 con 0x55c1aa89c130 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce17fa640 1 -- 172.21.15.145:0/388643707 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3cdc0cf340 con 0x7f3ce4102160 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fded2b7043ccf69f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3ce8e09640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 0x7f3ccc0a4cf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0963d0 tx=0x7f3cdc080040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3cca7fc640 1 -- 172.21.15.145:0/3469344420 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd802cb80 con 0x55c1aa89c130 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3cca7fc640 1 -- 172.21.15.145:0/3469344420 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd8005040 con 0x55c1aa89c130 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.699+0000 7f3cca7fc640 1 -- 172.21.15.145:0/3469344420 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8004900 con 0x55c1aa89c130 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- 172.21.15.145:0/3469344420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 msgr2=0x55c1aa8a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd8009560 comp rx=0 tx=0).stop 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- 172.21.15.145:0/3469344420 shutdown_connections 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8a0c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 --2- 172.21.15.145:0/3469344420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89b740 0x55c1aa89bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- 172.21.15.145:0/3469344420 >> 172.21.15.145:0/3469344420 conn(0x55c1aa863a60 msgr2=0x55c1aa8970b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- 172.21.15.145:0/3469344420 shutdown_connections 2024-03-28T18:38:05.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- 172.21.15.145:0/3469344420 wait complete. 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.700+0000 7f3ced458740 1 -- start start 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 Processor -- start 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89c130 0x55c1aa8a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8a41f0 con 0x55c1aa89b740 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa7bce90 con 0x55c1aa89c130 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8a4370 con 0x55c1aa8a12a0 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 10 client.0 _ll_register_callbacks cb 0x55c1aa514ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56648/0 (socket says 172.21.15.145:56648) 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 -- 172.21.15.145:0/3364227490 learned_addr learned my addr 172.21.15.145:0/3364227490 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89c130 0x55c1aa8a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 -- 172.21.15.145:0/3364227490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa8a12a0 msgr2=0x55c1aa8ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 -- 172.21.15.145:0/3364227490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89c130 msgr2=0x55c1aa8a61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89c130 0x55c1aa8a61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 -- 172.21.15.145:0/3364227490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1aa7c99d0 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40037f0 tx=0x7f3cd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1bab012a813c477 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3cc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40047a0 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3cd4009040 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.701+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cd8005040 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cd4004940 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 10 client.15435 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c1aa5152d0 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd8005040 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3cd4011aa0 con 0x55c1aa89b740 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 10 client.15435 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 10 client.15435 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:38:05.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 20 client.15435 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8b2480 0x55c1aa8b1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 4 client.15435 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 --> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c1aa80e160 con 0x55c1aa8b2480 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8b2480 0x55c1aa8b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3cc37fe640 10 client.15435 ms_handle_connect on v2:172.21.15.145:6840/2292179731 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.702+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8b2480 0x55c1aa8b1550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fca9eacd7c48dd35 server_cookie=186a83f69051a373 in_seq=0 out_seq=0 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.703+0000 7f3cc37fe640 1 -- 172.21.15.145:0/3364227490 <== mds.4 v2:172.21.15.145:6840/2292179731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c1aa80e160 con 0x55c1aa8b2480 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.703+0000 7f3cc37fe640 10 client.15435 handle_command_reply: tid=0 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.703+0000 7f3ced458740 1 client.15435 shutdown 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 msgr2=0x55c1aa8a5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40037f0 tx=0x7f3cd4004730 comp rx=0 tx=0).stop 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 shutdown_connections 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8b2480 0x55c1aa8b1550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa8a12a0 0x55c1aa8ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3cc37fe640 0 client.15435 ms_handle_reset on v2:172.21.15.145:6840/2292179731 2024-03-28T18:38:05.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa89c130 0x55c1aa8a61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 --2- 172.21.15.145:0/3364227490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8a5cb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 >> 172.21.15.145:0/3364227490 conn(0x55c1aa863a60 msgr2=0x55c1aa8962f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 shutdown_connections 2024-03-28T18:38:05.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 1 -- 172.21.15.145:0/3364227490 wait complete. 2024-03-28T18:38:05.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.704+0000 7f3ced458740 20 client.15435 trim_cache size 0 max 16384 2024-03-28T18:38:05.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 Processor -- start 2024-03-28T18:38:05.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 -- start start 2024-03-28T18:38:05.739 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b1ca0 con 0x55c1aa89b740 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b2300 con 0x55c1aa863a60 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ced458740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b2960 con 0x55c1aa89c130 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56654/0 (socket says 172.21.15.145:56654) 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 -- 172.21.15.145:0/2175264671 learned_addr learned my addr 172.21.15.145:0/2175264671 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 -- 172.21.15.145:0/2175264671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 msgr2=0x55c1aa8ba920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8ba920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 -- 172.21.15.145:0/2175264671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 msgr2=0x55c1aa8a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.740 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.734+0000 7f3ce3fff640 1 -- 172.21.15.145:0/2175264671 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1aa5df6b0 con 0x55c1aa89b740 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3cd802cfd0 tx=0x7f3cd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171ca7fa583e1745 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3cc9ffb640 1 -- 172.21.15.145:0/2175264671 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd802cce0 con 0x55c1aa89b740 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3cc9ffb640 1 -- 172.21.15.145:0/2175264671 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd804e600 con 0x55c1aa89b740 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3cc9ffb640 1 -- 172.21.15.145:0/2175264671 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8003850 con 0x55c1aa89b740 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 -- 172.21.15.145:0/2175264671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 msgr2=0x55c1aa8ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3cd802cfd0 tx=0x7f3cd8000bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 -- 172.21.15.145:0/2175264671 shutdown_connections 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89c130 0x55c1aa8ba920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89b740 0x55c1aa8ad180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 --2- 172.21.15.145:0/2175264671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 -- 172.21.15.145:0/2175264671 >> 172.21.15.145:0/2175264671 conn(0x55c1aa8b18c0 msgr2=0x55c1aa8a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.735+0000 7f3ced458740 1 -- 172.21.15.145:0/2175264671 shutdown_connections 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 -- 172.21.15.145:0/2175264671 wait complete. 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 -- start start 2024-03-28T18:38:05.741 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 Processor -- start 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 0x55c1aa8a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b3620 con 0x55c1aa89c130 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b2fc0 con 0x55c1aa863a60 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1aa8b3c80 con 0x55c1aa89b740 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ced458740 10 client.0 _ll_register_callbacks cb 0x7f3cd4009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 0x55c1aa8a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56668/0 (socket says 172.21.15.145:56668) 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ce37fe640 1 -- 172.21.15.145:0/25669822 learned_addr learned my addr 172.21.15.145:0/25669822 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.736+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 -- 172.21.15.145:0/25669822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 msgr2=0x55c1aa8a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.742 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 -- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 msgr2=0x55c1aa8ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 -- 172.21.15.145:0/25669822 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1aa8ace80 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ce3fff640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 0x55c1aa8a98e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55c1aa89ca10 tx=0x7f3cd802c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e607baca72eae6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8004240 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cd804e600 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3cd80043e0 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cd805e050 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ced458740 10 client.25300 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c1aa90dfa0 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd804e600 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.737+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f3cd8003af0 con 0x55c1aa89b740 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 10 client.25300 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:05.743 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 10 client.25300 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 20 client.25300 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8a12a0 0x55c1aa90e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 4 client.25300 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 --> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c1aa8084b0 con 0x55c1aa8a12a0 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8a12a0 0x55c1aa90e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ce37fe640 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8a12a0 0x55c1aa90e170 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6b93cc4cba45ea34 server_cookie=8af7bc94f11f1c29 in_seq=0 out_seq=0 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3cc37fe640 10 client.25300 ms_handle_connect on v2:172.21.15.145:6840/2292179731 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3cc37fe640 1 -- 172.21.15.145:0/25669822 <== mds.4 v2:172.21.15.145:6840/2292179731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c1aa8084b0 con 0x55c1aa8a12a0 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3cc37fe640 10 client.25300 handle_command_reply: tid=0 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 client.25300 shutdown 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 msgr2=0x55c1aa8a98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 0x55c1aa8a98e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55c1aa89ca10 tx=0x7f3cd802c3e0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 shutdown_connections 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:6840/2292179731,v1:172.21.15.145:6841/2292179731] conn(0x55c1aa8a12a0 0x55c1aa90e170 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c1aa89c130 0x55c1aa8ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.744 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3cc37fe640 0 client.25300 ms_handle_reset on v2:172.21.15.145:6840/2292179731 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c1aa89b740 0x55c1aa8a98e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 --2- 172.21.15.145:0/25669822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c1aa863a60 0x55c1aa8a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.738+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 >> 172.21.15.145:0/25669822 conn(0x55c1aa8b18c0 msgr2=0x55c1aa89a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 shutdown_connections 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3ced458740 1 -- 172.21.15.145:0/25669822 wait complete. 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3ced458740 20 client.25300 trim_cache size 0 max 16384 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 4, 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 14631, 2024-03-28T18:38:05.745 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 113.731783511, 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 95, 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 144.39548159500001 2024-03-28T18:38:05.746 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 msgr2=0x7f3ccc0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 0x7f3ccc0a4cf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0963d0 tx=0x7f3cdc080040 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 msgr2=0x7f3ce41a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00eb10 tx=0x7f3cdc00c2d0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 shutdown_connections 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce4102160 0x7f3ce41a1070 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce41017a0 0x7f3ce419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2830 0x7f3ccc0a4cf0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 --2- 172.21.15.145:0/388643707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce41005b0 0x7f3ce406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 >> 172.21.15.145:0/388643707 conn(0x7f3ce40fbb40 msgr2=0x7f3ce4104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 shutdown_connections 2024-03-28T18:38:05.747 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.739+0000 7f3cea86c640 1 -- 172.21.15.145:0/388643707 wait complete. 2024-03-28T18:38:05.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1838427118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b541005a0 msgr2=0x7f1b541009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1838427118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b541005a0 0x7f1b541009c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009ca0 tx=0x7f1b4402c100 comp rx=0 tx=0).stop 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1838427118 shutdown_connections 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1838427118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 0x7f1b541089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1838427118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b54101790 0x7f1b54101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1838427118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b541005a0 0x7f1b541009c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1838427118 >> 172.21.15.145:0/1838427118 conn(0x7f1b540fbb30 msgr2=0x7f1b540fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1838427118 shutdown_connections 2024-03-28T18:38:05.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1838427118 wait complete. 2024-03-28T18:38:05.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.854+0000 7f1b5a0b2640 1 Processor -- start 2024-03-28T18:38:05.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 -- start start 2024-03-28T18:38:05.856 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 0x7f1b54197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 0x7f1b5419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5410c200 con 0x7f1b54101790 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5410c080 con 0x7f1b541005a0 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b5a0b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5410c380 con 0x7f1b54102150 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 0x7f1b54197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56680/0 (socket says 172.21.15.145:56680) 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 -- 172.21.15.145:0/1158007194 learned_addr learned my addr 172.21.15.145:0/1158007194 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b53fff640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 0x7f1b5419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 -- 172.21.15.145:0/1158007194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 msgr2=0x7f1b5419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 0x7f1b5419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 -- 172.21.15.145:0/1158007194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 msgr2=0x7f1b54197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 0x7f1b54197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 -- 172.21.15.145:0/1158007194 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5419f640 con 0x7f1b54101790 2024-03-28T18:38:05.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b537fe640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 0x7f1b54197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.855+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.856+0000 7f1b50ff9640 1 -- 172.21.15.145:0/1158007194 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00cd80 con 0x7f1b54101790 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.856+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5419f930 con 0x7f1b54101790 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.856+0000 7f1b50ff9640 1 -- 172.21.15.145:0/1158007194 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b3c010910 con 0x7f1b54101790 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.856+0000 7f1b50ff9640 1 -- 172.21.15.145:0/1158007194 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c0188f0 con 0x7f1b54101790 2024-03-28T18:38:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.856+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5419fed0 con 0x7f1b54101790 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 Processor -- start 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 -- start start 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b50ff9640 1 -- 172.21.15.145:0/1158007194 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1b3c010ac0 con 0x7f1b54101790 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4395eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.864 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b43966730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b438c3c20 con 0x560b4395f130 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b4386ca40 con 0x560b439642a0 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b438bfcc0 con 0x560b4395e740 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56686/0 (socket says 172.21.15.145:56686) 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b53fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4395eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b537fe640 1 -- 172.21.15.145:0/1699442406 learned_addr learned my addr 172.21.15.145:0/1699442406 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.858+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b43966730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 -- 172.21.15.145:0/1699442406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 msgr2=0x560b43966730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b43966730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 -- 172.21.15.145:0/1699442406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 msgr2=0x560b43963c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 -- 172.21.15.145:0/1699442406 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b4388ef90 con 0x560b4395e740 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b537fe640 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b50ff9640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 0x7f1b400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b53fff640 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4395eb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009b80 tx=0x7f1b4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ad7c3d6abbb6f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b537fe640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 0x7f1b400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b49ffb640 1 -- 172.21.15.145:0/1699442406 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c004030 con 0x560b4395e740 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b50ff9640 1 -- 172.21.15.145:0/1158007194 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1b3c0cf1c0 con 0x7f1b54101790 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b49ffb640 1 -- 172.21.15.145:0/1699442406 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b4c005500 con 0x560b4395e740 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.859+0000 7f1b49ffb640 1 -- 172.21.15.145:0/1699442406 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c004dd0 con 0x560b4395e740 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- 172.21.15.145:0/1699442406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 msgr2=0x560b4395eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b537fe640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 0x7f1b400a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x560b4395f530 tx=0x7f1b44032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4395eb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009b80 tx=0x7f1b4c011810 comp rx=0 tx=0).stop 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- 172.21.15.145:0/1699442406 shutdown_connections 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b43966730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b43963c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/1699442406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4395eb60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- 172.21.15.145:0/1699442406 >> 172.21.15.145:0/1699442406 conn(0x560b43926a60 msgr2=0x560b4395a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- 172.21.15.145:0/1699442406 shutdown_connections 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- 172.21.15.145:0/1699442406 wait complete. 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- start start 2024-03-28T18:38:05.866 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 Processor -- start 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b4395f130 0x560b43969200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 0x560b43970240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b43967250 con 0x560b4395e740 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b4387fe90 con 0x560b4395f130 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b439673d0 con 0x560b439642a0 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.860+0000 7f1b5cd80740 10 client.0 _ll_register_callbacks cb 0x560b435d7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56696/0 (socket says 172.21.15.145:56696) 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b537fe640 1 -- 172.21.15.145:0/983628477 learned_addr learned my addr 172.21.15.145:0/983628477 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 0x560b43970240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b4395f130 0x560b43969200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 -- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b4395f130 msgr2=0x560b43969200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b4395f130 0x560b43969200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 -- 172.21.15.145:0/983628477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 msgr2=0x560b43968cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 -- 172.21.15.145:0/983628477 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b4388c9d0 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b537fe640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b53fff640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 0x560b43970240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c011d10 tx=0x7f1b4c0051f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2a4b0b30a0a04e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c028020 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b4c005500 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1b4c024620 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b4c033050 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b5cd80740 10 client.25312 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.861+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560b435d82d0 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b4c005500 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1b4c018070 con 0x560b439642a0 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 10 client.25312 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 10 client.25312 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 20 client.25312 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439751a0 0x560b43974450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 4 client.25312 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:05.868 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 --> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b438d1160 con 0x560b439751a0 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b537fe640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439751a0 0x560b43974450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b32ffd640 10 client.25312 ms_handle_connect on v2:172.21.15.164:6836/1634173775 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.862+0000 7f1b537fe640 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439751a0 0x560b43974450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d8acde6f17d0951 server_cookie=24c5fde9722a8b04 in_seq=0 out_seq=0 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b32ffd640 1 -- 172.21.15.145:0/983628477 <== mds.3 v2:172.21.15.164:6836/1634173775 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560b438d1160 con 0x560b439751a0 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b32ffd640 10 client.25312 handle_command_reply: tid=0 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 client.25312 shutdown 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 msgr2=0x560b43970240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 0x560b43970240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c011d10 tx=0x7f1b4c0051f0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 shutdown_connections 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439751a0 0x560b43974450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b439642a0 0x560b43970240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b4395f130 0x560b43969200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b32ffd640 0 client.25312 ms_handle_reset on v2:172.21.15.164:6836/1634173775 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/983628477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b43968cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 >> 172.21.15.145:0/983628477 conn(0x560b43926a60 msgr2=0x560b439592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 shutdown_connections 2024-03-28T18:38:05.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 1 -- 172.21.15.145:0/983628477 wait complete. 2024-03-28T18:38:05.870 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.863+0000 7f1b5cd80740 20 client.25312 trim_cache size 0 max 16384 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 Processor -- start 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 -- start start 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395f130 0x560b43970200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4397d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b439749c0 con 0x560b4395e740 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b43975020 con 0x560b439642a0 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b43975680 con 0x560b4395f130 2024-03-28T18:38:05.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56708/0 (socket says 172.21.15.145:56708) 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 -- 172.21.15.145:0/4283279791 learned_addr learned my addr 172.21.15.145:0/4283279791 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b53fff640 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4397d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395f130 0x560b43970200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 -- 172.21.15.145:0/4283279791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395f130 msgr2=0x560b43970200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395f130 0x560b43970200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 -- 172.21.15.145:0/4283279791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 msgr2=0x560b4397d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4397d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 -- 172.21.15.145:0/4283279791 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b436a26b0 con 0x560b4395e740 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b537fe640 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005e30 tx=0x7f1b44041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302c168c7c491ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.880+0000 7f1b497fa640 1 -- 172.21.15.145:0/4283279791 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44056020 con 0x560b4395e740 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b497fa640 1 -- 172.21.15.145:0/4283279791 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b440521e0 con 0x560b4395e740 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b497fa640 1 -- 172.21.15.145:0/4283279791 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44039040 con 0x560b4395e740 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- 172.21.15.145:0/4283279791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 msgr2=0x560b4395da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005e30 tx=0x7f1b44041000 comp rx=0 tx=0).stop 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- 172.21.15.145:0/4283279791 shutdown_connections 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4397d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395f130 0x560b43970200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/4283279791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395e740 0x560b4395da00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- 172.21.15.145:0/4283279791 >> 172.21.15.145:0/4283279791 conn(0x560b43926a60 msgr2=0x560b4395d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- 172.21.15.145:0/4283279791 shutdown_connections 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- 172.21.15.145:0/4283279791 wait complete. 2024-03-28T18:38:05.883 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.881+0000 7f1b5cd80740 1 -- start start 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 Processor -- start 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4396c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4396f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b43976340 con 0x560b4395f130 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b43975ce0 con 0x560b439642a0 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b439769a0 con 0x560b4395e740 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b5cd80740 10 client.0 _ll_register_callbacks cb 0x7f1b3c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4396c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56722/0 (socket says 172.21.15.145:56722) 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 -- 172.21.15.145:0/3844356005 learned_addr learned my addr 172.21.15.145:0/3844356005 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b53fff640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4396f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 -- 172.21.15.145:0/3844356005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 msgr2=0x560b4396c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4396c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 -- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 msgr2=0x560b4396f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4396f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 -- 172.21.15.145:0/3844356005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b4396ff40 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b537fe640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4396c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b52ffd640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0d2a40 tx=0x7f1b3c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fc4cd7ca4281c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.882+0000 7f1b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c09b1c0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1b3c0860b0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b440521e0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b3c0862b0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560b439d0c90 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b440521e0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1b3c09acb0 con 0x560b4395f130 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 10 client.15459 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 20 client.15459 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439d5380 0x560b439d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 4 client.15459 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 --> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560b438cb4b0 con 0x560b439d5380 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.883+0000 7f1b537fe640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439d5380 0x560b439d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b32ffd640 10 client.15459 ms_handle_connect on v2:172.21.15.164:6836/1634173775 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b537fe640 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439d5380 0x560b439d0fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aed0aba9d02c5da6 server_cookie=c0d88ce98eb78e4c in_seq=0 out_seq=0 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b32ffd640 1 -- 172.21.15.145:0/3844356005 <== mds.3 v2:172.21.15.164:6836/1634173775 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x560b438cb4b0 con 0x560b439d5380 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b32ffd640 10 client.15459 handle_command_reply: tid=0 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 client.15459 shutdown 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 msgr2=0x560b4396c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0d2a40 tx=0x7f1b3c009310 comp rx=0 tx=0).stop 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 shutdown_connections 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:6836/1634173775,v1:172.21.15.164:6837/1634173775] conn(0x560b439d5380 0x560b439d0fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.889 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560b439642a0 0x560b4396f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560b4395f130 0x560b4396c9d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b32ffd640 0 client.15459 ms_handle_reset on v2:172.21.15.164:6836/1634173775 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 --2- 172.21.15.145:0/3844356005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560b4395e740 0x560b4396c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 >> 172.21.15.145:0/3844356005 conn(0x560b43926a60 msgr2=0x560b43967800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 shutdown_connections 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 1 -- 172.21.15.145:0/3844356005 wait complete. 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.884+0000 7f1b5cd80740 20 client.15459 trim_cache size 0 max 16384 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 3, 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24461, 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:38:05.890 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 114.79739692, 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 95, 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 153.142331047 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 msgr2=0x7f1b400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 0x7f1b400a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x560b4395f530 tx=0x7f1b44032590 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 msgr2=0x7f1b54197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 shutdown_connections 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1b400a2800 0x7f1b400a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b54102150 0x7f1b5419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54101790 0x7f1b54197f60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 --2- 172.21.15.145:0/1158007194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541005a0 0x7f1b54197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 >> 172.21.15.145:0/1158007194 conn(0x7f1b540fbb30 msgr2=0x7f1b54068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 shutdown_connections 2024-03-28T18:38:05.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:05.885+0000 7f1b5a0b2640 1 -- 172.21.15.145:0/1158007194 wait complete. 2024-03-28T18:38:05.895 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1543375096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.895 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/405547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.895 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:05 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1286580902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T18:38:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1543375096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/405547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:05 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1286580902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 -- 172.21.15.145:0/1255965932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a781016d0 msgr2=0x7f5a78101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/1255965932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a781016d0 0x7f5a78101af0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009900 tx=0x7f5a6c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 -- 172.21.15.145:0/1255965932 shutdown_connections 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/1255965932 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a78103280 0x7f5a78109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/1255965932 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 0x7f5a78102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/1255965932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a781016d0 0x7f5a78101af0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 -- 172.21.15.145:0/1255965932 >> 172.21.15.145:0/1255965932 conn(0x7f5a780fcc40 msgr2=0x7f5a780ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 -- 172.21.15.145:0/1255965932 shutdown_connections 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.000+0000 7f5a7eae8640 1 -- 172.21.15.145:0/1255965932 wait complete. 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 Processor -- start 2024-03-28T18:38:06.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 -- start start 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a781016d0 0x7f5a781979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 0x7f5a78197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7810d320 con 0x7f5a78103280 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7810d1a0 con 0x7f5a781016d0 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7eae8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7810d4a0 con 0x7f5a781028c0 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56732/0 (socket says 172.21.15.145:56732) 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 -- 172.21.15.145:0/3109619094 learned_addr learned my addr 172.21.15.145:0/3109619094 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7c884640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a781016d0 0x7f5a781979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a77fff640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 0x7f5a78197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 -- 172.21.15.145:0/3109619094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 msgr2=0x7f5a78197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 0x7f5a78197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 -- 172.21.15.145:0/3109619094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a781016d0 msgr2=0x7f5a781979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a781016d0 0x7f5a781979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 -- 172.21.15.145:0/3109619094 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7819f610 con 0x7f5a78103280 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a7d085640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000efd0 tx=0x7f5a7000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.001+0000 7f5a75ffb640 1 -- 172.21.15.145:0/3109619094 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a70003b70 con 0x7f5a78103280 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.002+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7819f900 con 0x7f5a78103280 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.002+0000 7f5a75ffb640 1 -- 172.21.15.145:0/3109619094 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a70003d10 con 0x7f5a78103280 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.002+0000 7f5a75ffb640 1 -- 172.21.15.145:0/3109619094 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a700109f0 con 0x7f5a78103280 2024-03-28T18:38:06.004 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.002+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7819fe50 con 0x7f5a78103280 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a75ffb640 1 -- 172.21.15.145:0/3109619094 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5a70004020 con 0x7f5a78103280 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 Processor -- start 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 -- start start 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f19b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f21730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.010 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309e7ec20 con 0x563309f1a130 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309e27a40 con 0x563309f19740 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.004+0000 7f5a817af740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309e7acc0 con 0x563309f1f2a0 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f19b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a75ffb640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 0x7f5a680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56744/0 (socket says 172.21.15.145:56744) 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7c884640 1 -- 172.21.15.145:0/2071787127 learned_addr learned my addr 172.21.15.145:0/2071787127 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f21730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7d085640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 0x7f5a680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a75ffb640 1 -- 172.21.15.145:0/3109619094 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5a700cf280 con 0x7f5a78103280 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 -- 172.21.15.145:0/2071787127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 msgr2=0x563309f21730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f21730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 -- 172.21.15.145:0/2071787127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 msgr2=0x563309f1ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.011 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 -- 172.21.15.145:0/2071787127 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563309e49f90 con 0x563309f19740 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7c884640 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a77fff640 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f19b60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009730 tx=0x7f5a640096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30541faf04b4860 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a7d085640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 0x7f5a680a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5a700cca20 tx=0x7f5a700c62c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a62ffd640 1 -- 172.21.15.145:0/2071787127 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64004050 con 0x563309f19740 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a62ffd640 1 -- 172.21.15.145:0/2071787127 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a64005500 con 0x563309f19740 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.005+0000 7f5a62ffd640 1 -- 172.21.15.145:0/2071787127 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64004d00 con 0x563309f19740 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- 172.21.15.145:0/2071787127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 msgr2=0x563309f19b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f19b60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009730 tx=0x7f5a640096c0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- 172.21.15.145:0/2071787127 shutdown_connections 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f21730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f1ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 --2- 172.21.15.145:0/2071787127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f19b60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- 172.21.15.145:0/2071787127 >> 172.21.15.145:0/2071787127 conn(0x563309ee1a60 msgr2=0x563309f150b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- 172.21.15.145:0/2071787127 shutdown_connections 2024-03-28T18:38:06.012 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- 172.21.15.145:0/2071787127 wait complete. 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 -- start start 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.006+0000 7f5a817af740 1 Processor -- start 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1a130 0x563309f241c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f2b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f22250 con 0x563309f19740 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309e3ae90 con 0x563309f1f2a0 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f223d0 con 0x563309f1a130 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a817af740 10 client.0 _ll_register_callbacks cb 0x563309b92ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1a130 0x563309f241c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56756/0 (socket says 172.21.15.145:56756) 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 -- 172.21.15.145:0/225800573 learned_addr learned my addr 172.21.15.145:0/225800573 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f2b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 -- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1a130 msgr2=0x563309f241c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1a130 0x563309f241c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.013 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 -- 172.21.15.145:0/225800573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 msgr2=0x563309f2b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f2b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 -- 172.21.15.145:0/225800573 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563309e479d0 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a77fff640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5a64006010 tx=0x7f5a64005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7b71dc50f684f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a57fff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.007+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400f790 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5a6400f930 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a64005500 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a64036050 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563309b932d0 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a64005500 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5a6401fc10 con 0x563309f19740 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 10 client.15471 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:38:06.014 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 20 client.15471 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f30090 0x563309f2f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 4 client.15471 mds_command: new command op to 24469 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 --> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563309e8c160 con 0x563309f30090 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.008+0000 7f5a7c884640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f30090 0x563309f2f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a57fff640 10 client.15471 ms_handle_connect on v2:172.21.15.192:6836/2153191464 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a7c884640 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f30090 0x563309f2f360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eb77e21e6960bbfc server_cookie=6bf53e6eb3d84e6c in_seq=0 out_seq=0 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a57fff640 1 -- 172.21.15.145:0/225800573 <== mds.2 v2:172.21.15.192:6836/2153191464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563309e8c160 con 0x563309f30090 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a57fff640 10 client.15471 handle_command_reply: tid=0 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 client.15471 shutdown 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 msgr2=0x563309f23c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5a64006010 tx=0x7f5a64005850 comp rx=0 tx=0).stop 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 shutdown_connections 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f2b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1a130 0x563309f241c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f30090 0x563309f2f360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 --2- 172.21.15.145:0/225800573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f19740 0x563309f23c80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a57fff640 0 client.15471 ms_handle_reset on v2:172.21.15.192:6836/2153191464 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.009+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 >> 172.21.15.145:0/225800573 conn(0x563309ee1a60 msgr2=0x563309f142f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.010+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 shutdown_connections 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.010+0000 7f5a817af740 1 -- 172.21.15.145:0/225800573 wait complete. 2024-03-28T18:38:06.016 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.010+0000 7f5a817af740 20 client.15471 trim_cache size 0 max 16384 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 Processor -- start 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 -- start start 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f19740 0x563309f23f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f38530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f2f8b0 con 0x563309f1a130 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f2ff10 con 0x563309f1f2a0 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a817af740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f30570 con 0x563309f19740 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.026+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56770/0 (socket says 172.21.15.145:56770) 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 -- 172.21.15.145:0/289939530 learned_addr learned my addr 172.21.15.145:0/289939530 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a77fff640 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f19740 0x563309f23f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a77fff640 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f38530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 -- 172.21.15.145:0/289939530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f19740 msgr2=0x563309f23f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.032 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f19740 0x563309f23f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 -- 172.21.15.145:0/289939530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 msgr2=0x563309f38530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f38530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 -- 172.21.15.145:0/289939530 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563309c5d6b0 con 0x563309f1a130 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a7c884640 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5a78102720 tx=0x7f5a6c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38ce1a83e35ddb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a627fc640 1 -- 172.21.15.145:0/289939530 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c04b070 con 0x563309f1a130 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a627fc640 1 -- 172.21.15.145:0/289939530 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a6c02cd50 con 0x563309f1a130 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.027+0000 7f5a627fc640 1 -- 172.21.15.145:0/289939530 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c004920 con 0x563309f1a130 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- 172.21.15.145:0/289939530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 msgr2=0x563309f2b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5a78102720 tx=0x7f5a6c02e040 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- 172.21.15.145:0/289939530 shutdown_connections 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f1f2a0 0x563309f38530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f2b1c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- 172.21.15.145:0/289939530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f19740 0x563309f23f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- 172.21.15.145:0/289939530 >> 172.21.15.145:0/289939530 conn(0x563309ee1a60 msgr2=0x563309f23030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- 172.21.15.145:0/289939530 shutdown_connections 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- 172.21.15.145:0/289939530 wait complete. 2024-03-28T18:38:06.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- start start 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 Processor -- start 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f27980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f27ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f31230 con 0x563309f1a130 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f30bd0 con 0x563309f19740 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.028+0000 7f5a817af740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563309f31890 con 0x563309f1f2a0 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a817af740 10 client.0 _ll_register_callbacks cb 0x7f5a64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f27980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a77fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60160/0 (socket says 172.21.15.145:60160) 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a77fff640 1 -- 172.21.15.145:0/3588591125 learned_addr learned my addr 172.21.15.145:0/3588591125 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a77fff640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f27ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 -- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 msgr2=0x563309f27ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f27ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 -- 172.21.15.145:0/3588591125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 msgr2=0x563309f2aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 -- 172.21.15.145:0/3588591125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563309f28480 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a77fff640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a7c884640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f27980 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009900 tx=0x7f5a6c02ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a134160c4312ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a57fff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c04b040 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5a6c005380 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a6c02cd50 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a6c05c050 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a817af740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563309f8c130 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.029+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a6c02cd50 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5a6c057030 con 0x563309f1a130 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 10 client.15477 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 20 client.15477 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f90380 0x563309f8c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 4 client.15477 mds_command: new command op to 24469 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 --> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563309e864b0 con 0x563309f90380 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a77fff640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f90380 0x563309f8c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a57fff640 10 client.15477 ms_handle_connect on v2:172.21.15.192:6836/2153191464 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a77fff640 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f90380 0x563309f8c300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c825c3532de14cc8 server_cookie=1d2fb3055865efae in_seq=0 out_seq=0 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a57fff640 1 -- 172.21.15.145:0/3588591125 <== mds.2 v2:172.21.15.192:6836/2153191464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x563309e864b0 con 0x563309f90380 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a57fff640 10 client.15477 handle_command_reply: tid=0 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.030+0000 7f5a817af740 1 client.15477 shutdown 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 msgr2=0x563309f27980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f27980 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009900 tx=0x7f5a6c02ca00 comp rx=0 tx=0).stop 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 shutdown_connections 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563309f1f2a0 0x563309f27ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.192:6836/2153191464,v1:172.21.15.192:6837/2153191464] conn(0x563309f90380 0x563309f8c300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563309f1a130 0x563309f27980 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 --2- 172.21.15.145:0/3588591125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563309f19740 0x563309f2aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 >> 172.21.15.145:0/3588591125 conn(0x563309ee1a60 msgr2=0x563309f23030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a57fff640 0 client.15477 ms_handle_reset on v2:172.21.15.192:6836/2153191464 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 shutdown_connections 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 1 -- 172.21.15.145:0/3588591125 wait complete. 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a817af740 20 client.15477 trim_cache size 0 max 16384 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 2, 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24469, 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:38:06.037 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 115.94775295700001, 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 95, 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 156.27007425599999 2024-03-28T18:38:06.038 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 msgr2=0x7f5a680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 0x7f5a680a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5a700cca20 tx=0x7f5a700c62c0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 msgr2=0x7f5a7819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000efd0 tx=0x7f5a7000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 shutdown_connections 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a78103280 0x7f5a7819efa0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5a680a2800 0x7f5a680a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a781028c0 0x7f5a78197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 --2- 172.21.15.145:0/3109619094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a781016d0 0x7f5a781979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.031+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 >> 172.21.15.145:0/3109619094 conn(0x7f5a780fcc40 msgr2=0x7f5a78105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.032+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 shutdown_connections 2024-03-28T18:38:06.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.032+0000 7f5a7eae8640 1 -- 172.21.15.145:0/3109619094 wait complete. 2024-03-28T18:38:06.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:38:06.155 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1505566453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4102150 msgr2=0x7ff9e41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.155 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1505566453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4102150 0x7ff9e41089e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009ca0 tx=0x7ff9dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.155 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1505566453 shutdown_connections 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1505566453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4102150 0x7ff9e41089e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1505566453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e4101790 0x7ff9e4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1505566453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e41005a0 0x7ff9e41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1505566453 >> 172.21.15.145:0/1505566453 conn(0x7ff9e40fbb30 msgr2=0x7ff9e40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1505566453 shutdown_connections 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.155+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1505566453 wait complete. 2024-03-28T18:38:06.156 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 Processor -- start 2024-03-28T18:38:06.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1543375096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:38:06.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/405547726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:38:06.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:05 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1286580902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:38:06.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 -- start start 2024-03-28T18:38:06.158 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 0x7ff9e4196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 0x7ff9e419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e410c230 con 0x7ff9e4101790 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e410c0b0 con 0x7ff9e41005a0 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e9d87640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e410c3b0 con 0x7ff9e4102150 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56784/0 (socket says 172.21.15.145:56784) 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 0x7ff9e419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 0x7ff9e4196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.156+0000 7ff9e2ffd640 1 -- 172.21.15.145:0/1521803744 learned_addr learned my addr 172.21.15.145:0/1521803744 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 -- 172.21.15.145:0/1521803744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 msgr2=0x7ff9e4196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 0x7ff9e4196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 -- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 msgr2=0x7ff9e4196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.159 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 -- 172.21.15.145:0/1521803744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e419e480 con 0x7ff9e4102150 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 0x7ff9e4196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 0x7ff9e419de10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc02c630 tx=0x7ff9dc032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e0ff9640 1 -- 172.21.15.145:0/1521803744 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc041020 con 0x7ff9e4102150 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e419e710 con 0x7ff9e4102150 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e0ff9640 1 -- 172.21.15.145:0/1521803744 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9dc034cd0 con 0x7ff9e4102150 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e0ff9640 1 -- 172.21.15.145:0/1521803744 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc039040 con 0x7ff9e4102150 2024-03-28T18:38:06.160 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.157+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e419ec20 con 0x7ff9e4102150 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.159+0000 7ff9e0ff9640 1 -- 172.21.15.145:0/1521803744 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff9dc044020 con 0x7ff9e4102150 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.159+0000 7ff9ec971740 1 Processor -- start 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.159+0000 7ff9ec971740 1 -- start start 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 0x55a44a9c9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a926c20 con 0x55a44a9c1740 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a8cfa40 con 0x55a44a9c2130 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9ec971740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a922cc0 con 0x55a44a9c72a0 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56798/0 (socket says 172.21.15.145:56798) 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e2ffd640 1 -- 172.21.15.145:0/2120125168 learned_addr learned my addr 172.21.15.145:0/2120125168 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e0ff9640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 0x7ff9cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.167 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 0x55a44a9c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 0x7ff9cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2120125168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 msgr2=0x55a44a9c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 0x55a44a9c9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2120125168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 msgr2=0x55a44a9c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2120125168 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a44a8f1f90 con 0x55a44a9c2130 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 0x55a44a9c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e0ff9640 1 -- 172.21.15.145:0/1521803744 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff9dc0f0bd0 con 0x7ff9e4102150 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9c6c30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009fa0 tx=0x7ff9d40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3963434ce222996c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9e3fff640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 0x7ff9cc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ee370 tx=0x7ff9dc0b7c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.160+0000 7ff9c9ffb640 1 -- 172.21.15.145:0/2120125168 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4004030 con 0x55a44a9c2130 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9c9ffb640 1 -- 172.21.15.145:0/2120125168 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d40041d0 con 0x55a44a9c2130 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9c9ffb640 1 -- 172.21.15.145:0/2120125168 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400f890 con 0x55a44a9c2130 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 -- 172.21.15.145:0/2120125168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 msgr2=0x55a44a9c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.168 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9c6c30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009fa0 tx=0x7ff9d40118d0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 -- 172.21.15.145:0/2120125168 shutdown_connections 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c72a0 0x55a44a9c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9c6c30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2120125168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 -- 172.21.15.145:0/2120125168 >> 172.21.15.145:0/2120125168 conn(0x55a44a989a60 msgr2=0x55a44a9bd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 -- 172.21.15.145:0/2120125168 shutdown_connections 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.161+0000 7ff9ec971740 1 -- 172.21.15.145:0/2120125168 wait complete. 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 -- start start 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 Processor -- start 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 0x55a44a9cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9ca250 con 0x55a44a9c1740 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a8e2e90 con 0x55a44a9c72a0 2024-03-28T18:38:06.169 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9ca3d0 con 0x55a44a9c2130 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9ec971740 10 client.0 _ll_register_callbacks cb 0x55a44a63aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.162+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56806/0 (socket says 172.21.15.145:56806) 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 0x55a44a9cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e2ffd640 1 -- 172.21.15.145:0/700850868 learned_addr learned my addr 172.21.15.145:0/700850868 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 -- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 msgr2=0x55a44a9d3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 -- 172.21.15.145:0/700850868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 msgr2=0x55a44a9cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 -- 172.21.15.145:0/700850868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a44a8ef9d0 con 0x55a44a9c2130 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 0x55a44a9cc1c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55a44a9be000 tx=0x7ff9d4003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4541ca64af973c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.170 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4005040 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9d40041d0 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff9d4016040 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9d4032050 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9ec971740 10 client.25345 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a44a63b2d0 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.163+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d40041d0 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff9d400f610 con 0x55a44a9c2130 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 10 client.25345 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 10 client.25345 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 20 client.25345 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44a9d8100 0x55a44a9d7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 4 client.25345 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 --> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a44a934160 con 0x55a44a9d8100 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44a9d8100 0x55a44a9d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.171 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9c2ffd640 10 client.25345 ms_handle_connect on v2:172.21.15.164:6838/644460294 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.164+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44a9d8100 0x55a44a9d7390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acfef23f77ab437f server_cookie=3b8db1ef3903be1e in_seq=0 out_seq=0 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/700850868 <== mds.1 v2:172.21.15.164:6838/644460294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a44a934160 con 0x55a44a9d8100 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9c2ffd640 10 client.25345 handle_command_reply: tid=0 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 client.25345 shutdown 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 msgr2=0x55a44a9cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 0x55a44a9cc1c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55a44a9be000 tx=0x7ff9d4003d60 comp rx=0 tx=0).stop 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 shutdown_connections 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44a9d8100 0x55a44a9d7390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c2130 0x55a44a9cc1c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 --2- 172.21.15.145:0/700850868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c1740 0x55a44a9cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9c2ffd640 0 client.25345 ms_handle_reset on v2:172.21.15.164:6838/644460294 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.165+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 >> 172.21.15.145:0/700850868 conn(0x55a44a989a60 msgr2=0x55a44a9bc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.166+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 shutdown_connections 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.166+0000 7ff9ec971740 1 -- 172.21.15.145:0/700850868 wait complete. 2024-03-28T18:38:06.172 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.166+0000 7ff9ec971740 20 client.25345 trim_cache size 0 max 16384 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 Processor -- start 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 -- start start 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c72a0 0x55a44a9cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d7920 con 0x55a44a9c72a0 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d7f80 con 0x55a44a9c2130 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9ec971740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d85e0 con 0x55a44a9c1740 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c72a0 0x55a44a9cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.183 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60212/0 (socket says 172.21.15.145:60212) 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9c0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:48278/0 (socket says 172.21.15.145:48278) 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.182+0000 7ff9e37fe640 1 -- 172.21.15.145:0/890342555 learned_addr learned my addr 172.21.15.145:0/890342555 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 -- 172.21.15.145:0/890342555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 msgr2=0x55a44a9c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 -- 172.21.15.145:0/890342555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c72a0 msgr2=0x55a44a9cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c72a0 0x55a44a9cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 -- 172.21.15.145:0/890342555 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a44a9d7700 con 0x55a44a9c2130 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400fa80 tx=0x7ff9d400ffd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33190d4b170bf064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9c97fa640 1 -- 172.21.15.145:0/890342555 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4000e60 con 0x55a44a9c2130 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9c97fa640 1 -- 172.21.15.145:0/890342555 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4029070 con 0x55a44a9c2130 2024-03-28T18:38:06.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.183+0000 7ff9c97fa640 1 -- 172.21.15.145:0/890342555 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400f4e0 con 0x55a44a9c2130 2024-03-28T18:38:06.188 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- 172.21.15.145:0/890342555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 msgr2=0x55a44a9cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400fa80 tx=0x7ff9d400ffd0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- 172.21.15.145:0/890342555 shutdown_connections 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c72a0 0x55a44a9cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c2130 0x55a44a9cb890 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- 172.21.15.145:0/890342555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- 172.21.15.145:0/890342555 >> 172.21.15.145:0/890342555 conn(0x55a44a989a60 msgr2=0x55a44a9c02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- 172.21.15.145:0/890342555 shutdown_connections 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- 172.21.15.145:0/890342555 wait complete. 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- start start 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 Processor -- start 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d92a0 con 0x55a44a9c2130 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d8c40 con 0x55a44a9c72a0 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a44a9d9900 con 0x55a44a9c1740 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9ec971740 10 client.0 _ll_register_callbacks cb 0x7ff9d8009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9e37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56820/0 (socket says 172.21.15.145:56820) 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.184+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2913485670 learned_addr learned my addr 172.21.15.145:0/2913485670 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2913485670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 msgr2=0x55a44a9d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9d1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 msgr2=0x55a44a9d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 -- 172.21.15.145:0/2913485670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a44a7056b0 con 0x55a44a9c2130 2024-03-28T18:38:06.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9e37fe640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4011fd0 tx=0x7ff9d4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdd52c46f583e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400fda0 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9d4029070 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff9d400f4e0 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9d400f6e0 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9ec971740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a44aa35550 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.185+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4029070 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff9d4004a00 con 0x55a44a9c2130 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 10 client.15495 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 20 client.15495 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44aa39850 0x55a44aa35780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 4 client.15495 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 --> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a44a92e4b0 con 0x55a44aa39850 2024-03-28T18:38:06.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44aa39850 0x55a44aa35780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9c2ffd640 10 client.15495 ms_handle_connect on v2:172.21.15.164:6838/644460294 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9e2ffd640 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44aa39850 0x55a44aa35780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27429cf76f94b19d server_cookie=58ea2fc58a254164 in_seq=0 out_seq=0 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9c2ffd640 1 -- 172.21.15.145:0/2913485670 <== mds.1 v2:172.21.15.164:6838/644460294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a44a92e4b0 con 0x55a44aa39850 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9c2ffd640 10 client.15495 handle_command_reply: tid=0 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.186+0000 7ff9ec971740 1 client.15495 shutdown 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 msgr2=0x55a44a9d1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4011fd0 tx=0x7ff9d4009b30 comp rx=0 tx=0).stop 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 shutdown_connections 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:6838/644460294,v1:172.21.15.164:6839/644460294] conn(0x55a44aa39850 0x55a44aa35780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a44a9c72a0 0x55a44a9d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a44a9c2130 0x55a44a9d1b40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9c2ffd640 0 client.15495 ms_handle_reset on v2:172.21.15.164:6838/644460294 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 --2- 172.21.15.145:0/2913485670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a44a9c1740 0x55a44a9d1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 >> 172.21.15.145:0/2913485670 conn(0x55a44a989a60 msgr2=0x55a44a9cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 shutdown_connections 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 1 -- 172.21.15.145:0/2913485670 wait complete. 2024-03-28T18:38:06.192 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9ec971740 20 client.15495 trim_cache size 0 max 16384 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 1, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24491, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 117.113709075, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 95, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:38:06.193 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 141.93725757600001 2024-03-28T18:38:06.194 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:38:06.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 msgr2=0x7ff9cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.194 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 0x7ff9cc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ee370 tx=0x7ff9dc0b7c80 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 msgr2=0x7ff9e419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 0x7ff9e419de10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc02c630 tx=0x7ff9dc032f20 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 shutdown_connections 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff9cc0a2800 0x7ff9cc0a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9e4102150 0x7ff9e419de10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9e4101790 0x7ff9e4196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 --2- 172.21.15.145:0/1521803744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9e41005a0 0x7ff9e4196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 >> 172.21.15.145:0/1521803744 conn(0x7ff9e40fbb30 msgr2=0x7ff9e4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.187+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 shutdown_connections 2024-03-28T18:38:06.195 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.188+0000 7ff9e9d87640 1 -- 172.21.15.145:0/1521803744 wait complete. 2024-03-28T18:38:06.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:38:06.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 -- 172.21.15.145:0/1664803110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda781021c0 msgr2=0x7fda781045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 --2- 172.21.15.145:0/1664803110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda781021c0 0x7fda781045b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fda68009860 tx=0x7fda6802bb70 comp rx=0 tx=0).stop 2024-03-28T18:38:06.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 -- 172.21.15.145:0/1664803110 shutdown_connections 2024-03-28T18:38:06.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 --2- 172.21.15.145:0/1664803110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78107420 0x7fda78109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 --2- 172.21.15.145:0/1664803110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda78104af0 0x7fda78106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 --2- 172.21.15.145:0/1664803110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda781021c0 0x7fda781045b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 -- 172.21.15.145:0/1664803110 >> 172.21.15.145:0/1664803110 conn(0x7fda780fbb30 msgr2=0x7fda780fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 -- 172.21.15.145:0/1664803110 shutdown_connections 2024-03-28T18:38:06.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.303+0000 7fda7e340640 1 -- 172.21.15.145:0/1664803110 wait complete. 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 Processor -- start 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 -- start start 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78104af0 0x7fda78197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda78107420 0x7fda7819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7810d310 con 0x7fda781021c0 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7810d190 con 0x7fda78104af0 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda7e340640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7810d490 con 0x7fda78107420 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56834/0 (socket says 172.21.15.145:56834) 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda77fff640 1 -- 172.21.15.145:0/2219652182 learned_addr learned my addr 172.21.15.145:0/2219652182 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.304+0000 7fda777fe640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78104af0 0x7fda78197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda78107420 0x7fda7819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 -- 172.21.15.145:0/2219652182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda78107420 msgr2=0x7fda7819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda78107420 0x7fda7819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 -- 172.21.15.145:0/2219652182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78104af0 msgr2=0x7fda78197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78104af0 0x7fda78197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 -- 172.21.15.145:0/2219652182 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7819f630 con 0x7fda781021c0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda77fff640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda68005940 tx=0x7fda68032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda757fa640 1 -- 172.21.15.145:0/2219652182 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68041020 con 0x7fda781021c0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7819f8c0 con 0x7fda781021c0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda757fa640 1 -- 172.21.15.145:0/2219652182 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda6802c9f0 con 0x7fda781021c0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda757fa640 1 -- 172.21.15.145:0/2219652182 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68039040 con 0x7fda781021c0 2024-03-28T18:38:06.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.305+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7819fe00 con 0x7fda781021c0 2024-03-28T18:38:06.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.307+0000 7fda757fa640 1 -- 172.21.15.145:0/2219652182 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fda68044020 con 0x7fda781021c0 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.307+0000 7fda80ffd740 1 Processor -- start 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 -- start start 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e4a740 0x55fe93e4ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e4fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93dafc20 con 0x55fe93e502a0 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93d58a40 con 0x55fe93e4a740 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda80ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93dabcc0 con 0x55fe93e4b130 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda757fa640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 0x7fda600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda77fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e4fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda7c8dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e4a740 0x55fe93e4ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56850/0 (socket says 172.21.15.145:56850) 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 -- 172.21.15.145:0/1048519922 learned_addr learned my addr 172.21.15.145:0/1048519922 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 0x7fda600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda757fa640 1 -- 172.21.15.145:0/2219652182 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fda680f0760 con 0x7fda781021c0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 -- 172.21.15.145:0/1048519922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 msgr2=0x55fe93e4fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e4fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 -- 172.21.15.145:0/1048519922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e4a740 msgr2=0x55fe93e4ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e4a740 0x55fe93e4ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 -- 172.21.15.145:0/1048519922 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe93d7af90 con 0x55fe93e502a0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda77fff640 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e4fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda777fe640 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009c60 tx=0x7fda6c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0819d2c99245ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.308+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 0x7fda600a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fe93e4b990 tx=0x7fda700095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda5e7fc640 1 -- 172.21.15.145:0/1048519922 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c004230 con 0x55fe93e502a0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda5e7fc640 1 -- 172.21.15.145:0/1048519922 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda6c0043d0 con 0x55fe93e502a0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda5e7fc640 1 -- 172.21.15.145:0/1048519922 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c004d70 con 0x55fe93e502a0 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 -- 172.21.15.145:0/1048519922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 msgr2=0x55fe93e52730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009c60 tx=0x7fda6c011810 comp rx=0 tx=0).stop 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 -- 172.21.15.145:0/1048519922 shutdown_connections 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e502a0 0x55fe93e52730 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e4fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1048519922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e4a740 0x55fe93e4ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 -- 172.21.15.145:0/1048519922 >> 172.21.15.145:0/1048519922 conn(0x55fe93e12a60 msgr2=0x55fe93e460b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 -- 172.21.15.145:0/1048519922 shutdown_connections 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.309+0000 7fda80ffd740 1 -- 172.21.15.145:0/1048519922 wait complete. 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 -- start start 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 Processor -- start 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e54c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e53250 con 0x55fe93e4b130 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93d6be90 con 0x55fe93e502a0 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e533d0 con 0x55fe93e4a740 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda80ffd740 10 client.0 _ll_register_callbacks cb 0x55fe93ac3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56864/0 (socket says 172.21.15.145:56864) 2024-03-28T18:38:06.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda777fe640 1 -- 172.21.15.145:0/2483868235 learned_addr learned my addr 172.21.15.145:0/2483868235 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.310+0000 7fda77fff640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e54c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 -- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 msgr2=0x55fe93e54c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e54c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 -- 172.21.15.145:0/2483868235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 msgr2=0x55fe93e5c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 -- 172.21.15.145:0/2483868235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe93d789d0 con 0x55fe93e4b130 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda77fff640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e54c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda777fe640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda6c011d10 tx=0x7fda6c019ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d0e0a967580480 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda577fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c028020 con 0x55fe93e4b130 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fda6c024620 con 0x55fe93e4b130 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda6c0043d0 con 0x55fe93e4b130 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda6c033050 con 0x55fe93e4b130 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda80ffd740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe93ac42d0 con 0x55fe93e4b130 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.311+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda6c0043d0 con 0x55fe93e4b130 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fda6c018070 con 0x55fe93e4b130 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 10 client.15507 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 20 client.15507 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93e61190 0x55fe93e603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 4 client.15507 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe93dbd160 con 0x55fe93e61190 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda77fff640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93e61190 0x55fe93e603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda577fe640 10 client.15507 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.312+0000 7fda77fff640 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93e61190 0x55fe93e603b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662bb536665eb4a7 server_cookie=6f58a78b8d5e8b48 in_seq=0 out_seq=0 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda577fe640 1 -- 172.21.15.145:0/2483868235 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe93dbd160 con 0x55fe93e61190 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda577fe640 10 client.15507 handle_command_reply: tid=0 2024-03-28T18:38:06.318 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 client.15507 shutdown 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 msgr2=0x55fe93e55160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda6c011d10 tx=0x7fda6c019ac0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 shutdown_connections 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93e61190 0x55fe93e603b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e55160 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda577fe640 0 client.15507 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2483868235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e54c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 >> 172.21.15.145:0/2483868235 conn(0x55fe93e12a60 msgr2=0x55fe93e452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 shutdown_connections 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 1 -- 172.21.15.145:0/2483868235 wait complete. 2024-03-28T18:38:06.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.313+0000 7fda80ffd740 20 client.15507 trim_cache size 0 max 16384 2024-03-28T18:38:06.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 Processor -- start 2024-03-28T18:38:06.334 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 -- start start 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e5c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e69630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e609b0 con 0x55fe93e4a740 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e61010 con 0x55fe93e502a0 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda80ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e61670 con 0x55fe93e4b130 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56876/0 (socket says 172.21.15.145:56876) 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 -- 172.21.15.145:0/1472153664 learned_addr learned my addr 172.21.15.145:0/1472153664 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda777fe640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e5c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e69630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 -- 172.21.15.145:0/1472153664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 msgr2=0x55fe93e5c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e5c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 -- 172.21.15.145:0/1472153664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 msgr2=0x55fe93e69630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e69630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 -- 172.21.15.145:0/1472153664 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe93b8e6b0 con 0x55fe93e4a740 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e69630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda77fff640 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fda680b77f0 tx=0x7fda6808fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29b97671f78fef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda5dffb640 1 -- 172.21.15.145:0/1472153664 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6809b040 con 0x55fe93e4a740 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda5dffb640 1 -- 172.21.15.145:0/1472153664 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda680bd450 con 0x55fe93e4a740 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.330+0000 7fda5dffb640 1 -- 172.21.15.145:0/1472153664 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda680bb050 con 0x55fe93e4a740 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- 172.21.15.145:0/1472153664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 msgr2=0x55fe93e49a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fda680b77f0 tx=0x7fda6808fa80 comp rx=0 tx=0).stop 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- 172.21.15.145:0/1472153664 shutdown_connections 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e69630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4b130 0x55fe93e5c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 --2- 172.21.15.145:0/1472153664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4a740 0x55fe93e49a00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- 172.21.15.145:0/1472153664 >> 172.21.15.145:0/1472153664 conn(0x55fe93e12a60 msgr2=0x55fe93e492f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- 172.21.15.145:0/1472153664 shutdown_connections 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- 172.21.15.145:0/1472153664 wait complete. 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.331+0000 7fda80ffd740 1 -- start start 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 Processor -- start 2024-03-28T18:38:06.336 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e58490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e62330 con 0x55fe93e4b130 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e61cd0 con 0x55fe93e502a0 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe93e62990 con 0x55fe93e4a740 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 10 client.0 _ll_register_callbacks cb 0x7fda70009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda77fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e58490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56878/0 (socket says 172.21.15.145:56878) 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 -- 172.21.15.145:0/2734259626 learned_addr learned my addr 172.21.15.145:0/2734259626 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda7c8dd640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 -- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 msgr2=0x55fe93e58490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e58490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 -- 172.21.15.145:0/2734259626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 msgr2=0x55fe93e5b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 -- 172.21.15.145:0/2734259626 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe93e5bef0 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda777fe640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0197f0 tx=0x7fda6c0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76be5a4bbb69a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda577fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c03c060 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fda6c0051b0 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.332+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda680bd450 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda6c043050 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 10 client.15513 fetch_fsmap learned FSMap version 28 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe93ebccd0 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda680bd450 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fda6c018050 con 0x55fe93e4b130 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 10 client.15513 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 10 client.15513 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 20 client.15513 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93ec1230 0x55fe93ebcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 4 client.15513 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:38:06.338 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe93db74b0 con 0x55fe93ec1230 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.333+0000 7fda77fff640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93ec1230 0x55fe93ebcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda577fe640 10 client.15513 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda77fff640 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93ec1230 0x55fe93ebcea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f86c1dac2273aa52 server_cookie=cd5db41dd044cdd1 in_seq=0 out_seq=0 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda577fe640 1 -- 172.21.15.145:0/2734259626 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fe93db74b0 con 0x55fe93ec1230 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda577fe640 10 client.15513 handle_command_reply: tid=0 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 client.15513 shutdown 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 msgr2=0x55fe93e589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0197f0 tx=0x7fda6c0096d0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 shutdown_connections 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fe93ec1230 0x55fe93ebcea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe93e502a0 0x55fe93e5b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fe93e4b130 0x55fe93e589d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda577fe640 0 client.15513 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 --2- 172.21.15.145:0/2734259626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fe93e4a740 0x55fe93e58490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 >> 172.21.15.145:0/2734259626 conn(0x55fe93e12a60 msgr2=0x55fe93e49b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 shutdown_connections 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 1 -- 172.21.15.145:0/2734259626 wait complete. 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.334+0000 7fda80ffd740 20 client.15513 trim_cache size 0 max 16384 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "cluster_fsid": "b991dbde-ed30-11ee-95d0-87774f69a715", 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "whoami": 0, 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 24496, 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "want_state": "up:active", 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "state": "up:active", 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs", 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "rank_uptime": 135.41577227400001, 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "mdsmap_epoch": 28, 2024-03-28T18:38:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch": 95, 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_epoch_barrier": 91, 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "uptime": 150.81894914599999 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 msgr2=0x7fda600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 0x7fda600a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fe93e4b990 tx=0x7fda700095c0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 msgr2=0x7fda78197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda68005940 tx=0x7fda68032470 comp rx=0 tx=0).stop 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 shutdown_connections 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fda600a2800 0x7fda600a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda78107420 0x7fda7819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda78104af0 0x7fda78197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 --2- 172.21.15.145:0/2219652182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda781021c0 0x7fda78197a10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 >> 172.21.15.145:0/2219652182 conn(0x7fda780fbb30 msgr2=0x7fda78102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 shutdown_connections 2024-03-28T18:38:06.342 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:06.335+0000 7fda7e340640 1 -- 172.21.15.145:0/2219652182 wait complete. 2024-03-28T18:38:06.349 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T18:38:06.364 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T18:38:06.364 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T18:38:06.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:06.364 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-28T18:38:06.379 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T18:38:06.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:06.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T18:38:06.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:06.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-03-28T18:38:06.543 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T18:38:06.543 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:38:06.543 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:06.543 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T18:38:06.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:06 smithi145 ceph-mon[32151]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T18:38:06.794 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:38:06.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:06 smithi164 ceph-mon[36248]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T18:38:07.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:06 smithi192 ceph-mon[36589]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 -- 172.21.15.145:0/1275624243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374106370 msgr2=0x7f037410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 --2- 172.21.15.145:0/1275624243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374106370 0x7f037410df40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f036c00b0d0 tx=0x7f036c00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 -- 172.21.15.145:0/1275624243 shutdown_connections 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 --2- 172.21.15.145:0/1275624243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374106370 0x7f037410df40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 --2- 172.21.15.145:0/1275624243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101320 0x7f0374101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 --2- 172.21.15.145:0/1275624243 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374101cd0 0x7f0374105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 -- 172.21.15.145:0/1275624243 >> 172.21.15.145:0/1275624243 conn(0x7f03740770e0 msgr2=0x7f03740774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:08.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 -- 172.21.15.145:0/1275624243 shutdown_connections 2024-03-28T18:38:08.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.493+0000 7f0378d94700 1 -- 172.21.15.145:0/1275624243 wait complete. 2024-03-28T18:38:08.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 Processor -- start 2024-03-28T18:38:08.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 -- start start 2024-03-28T18:38:08.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101cd0 0x7f037419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 0x7f0374198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374111380 con 0x7f0374101320 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374111200 con 0x7f0374106370 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0378d94700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374111500 con 0x7f0374101cd0 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56890/0 (socket says 172.21.15.145:56890) 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 -- 172.21.15.145:0/1188279700 learned_addr learned my addr 172.21.15.145:0/1188279700 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0371d9b700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101cd0 0x7f037419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0372d9d700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 0x7f0374198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 -- 172.21.15.145:0/1188279700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101cd0 msgr2=0x7f037419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101cd0 0x7f037419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 -- 172.21.15.145:0/1188279700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 msgr2=0x7f0374198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 0x7f0374198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.496 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f037259c700 1 -- 172.21.15.145:0/1188279700 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374198fe0 con 0x7f0374101320 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.494+0000 7f0372d9d700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 0x7f0374198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f037259c700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f036400e9c0 tx=0x7f036400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400cbd0 con 0x7f0374101320 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f036400ee60 con 0x7f0374101320 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03741992d0 con 0x7f0374101320 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364005630 con 0x7f0374101320 2024-03-28T18:38:08.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.495+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03741a55b0 con 0x7f0374101320 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.496+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0374102770 con 0x7f0374101320 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.497+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0364004750 con 0x7f0374101320 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.498+0000 7f03637fe700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 0x7f035c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.498+0000 7f0371d9b700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 0x7f035c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.498+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f03640ceff0 con 0x7f0374101320 2024-03-28T18:38:08.498 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.498+0000 7f0371d9b700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 0x7f035c0a4dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0374199ec0 tx=0x7f0368009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:08.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.504+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f036409bce0 con 0x7f0374101320 2024-03-28T18:38:08.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.753+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0374199c30 con 0x7f0374101320 2024-03-28T18:38:08.754 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.754+0000 7f03637fe700 1 -- 172.21.15.145:0/1188279700 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7f036409b5e0 con 0x7f0374101320 2024-03-28T18:38:08.754 INFO:teuthology.orchestra.run.smithi145.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T18:38:08.755 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:08 smithi145 ceph-mon[32151]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:38:08.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 msgr2=0x7f035c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:08.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 0x7f035c0a4dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0374199ec0 tx=0x7f0368009260 comp rx=0 tx=0).stop 2024-03-28T18:38:08.756 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 msgr2=0x7f037419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f036400e9c0 tx=0x7f036400ecd0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 shutdown_connections 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f035c0a2910 0x7f035c0a4dd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374106370 0x7f0374198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0374101320 0x7f037419e630 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 --2- 172.21.15.145:0/1188279700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0374101cd0 0x7f037419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 >> 172.21.15.145:0/1188279700 conn(0x7f03740770e0 msgr2=0x7f037406a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 shutdown_connections 2024-03-28T18:38:08.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:08.756+0000 7f0378d94700 1 -- 172.21.15.145:0/1188279700 wait complete. 2024-03-28T18:38:08.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:08 smithi164 ceph-mon[36248]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:38:09.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:08 smithi192 ceph-mon[36589]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi145.front.sepia.ceph.com 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:38:09.471 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:38:09.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.472 DEBUG:teuthology.orchestra.run.smithi145:> ip addr 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:66 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f0 2024-03-28T18:38:09.488 INFO:teuthology.orchestra.run.smithi145.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:67 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f1 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:90 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:91 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.21.15.145/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft 42110sec preferred_lft 42110sec 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:38:09.489 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:38:09.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.489 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr flush dev ceph-brx 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set ceph-brx up 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:38:09.490 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:38:09.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.601 DEBUG:teuthology.orchestra.run.smithi145:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:38:09.628 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-03-28T18:38:09.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.629 DEBUG:teuthology.orchestra.run.smithi145:> route 2024-03-28T18:38:09.686 INFO:teuthology.orchestra.run.smithi145.stdout:Kernel IP routing table 2024-03-28T18:38:09.686 INFO:teuthology.orchestra.run.smithi145.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:38:09.686 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T18:38:09.686 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T18:38:09.687 INFO:teuthology.orchestra.run.smithi145.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:38:09.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T18:38:09.687 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:38:09.751 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:09 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1188279700' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.784 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-28T18:38:09.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.839 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list-id 2024-03-28T18:38:09.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.893 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:38:09.893 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:38:09.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:38:09.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T18:38:09.893 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:38:09.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:09 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1188279700' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:38:09.978 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T18:38:09.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:09.978 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:38:09.978 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:38:09.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T18:38:09.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:38:09.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T18:38:09.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T18:38:09.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T18:38:09.979 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:38:10.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.068 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-28T18:38:10.068 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-28T18:38:10.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set brx.0 up 2024-03-28T18:38:10.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:38:10.069 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-28T18:38:10.109 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:38:10.109 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:10.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:09 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1188279700' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:38:10.168 INFO:teuthology.orchestra.run.smithi145.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:38:10.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:38:10.168 DEBUG:teuthology.orchestra.run.smithi145:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:10.223 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T18:38:10.223 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T18:38:10.223 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/76215e0f-6dde-4a84-9c7d-3d313c968b82 2024-03-28T18:38:10.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/76215e0f-6dde-4a84-9c7d-3d313c968b82 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T18:38:10.312 INFO:teuthology.orchestra.run.smithi145.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T18:38:10.312 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T18:38:10.476 INFO:teuthology.orchestra.run.smithi145.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T18:38:10.476 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T18:38:10.476 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: resolved to: "172.21.15.145:3300,172.21.15.164:3300,172.21.15.192:3300" 2024-03-28T18:38:10.476 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: trying mount with new device syntax: 0@b991dbde-ed30-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/76215e0f-6dde-4a84-9c7d-3d313c968b82 2024-03-28T18:38:10.476 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.145:3300/172.21.15.164:3300/172.21.15.192:3300" will pass to kernel 2024-03-28T18:38:10.477 INFO:teuthology.orchestra.run.smithi145.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T18:38:10.477 INFO:teuthology.orchestra.run.smithi145.stdout: You just mounted a file system that supports labels which does not 2024-03-28T18:38:10.477 INFO:teuthology.orchestra.run.smithi145.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T18:38:10.477 INFO:teuthology.orchestra.run.smithi145.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T18:38:10.477 INFO:teuthology.orchestra.run.smithi145.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T18:38:10.479 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T18:38:10.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:10.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.510 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:38:10.565 INFO:teuthology.orchestra.run.smithi145.stdout:client15522 2024-03-28T18:38:10.565 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:38:10.566 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:38:10.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.621 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:38:10.677 INFO:teuthology.orchestra.run.smithi145.stdout:client15522 2024-03-28T18:38:10.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.677 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:38:10.734 INFO:teuthology.orchestra.run.smithi145.stdout:client15522 2024-03-28T18:38:10.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:38:10.734 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/b991dbde-ed30-11ee-95d0-87774f69a715.client15522/status') 2024-03-28T18:38:10.801 INFO:teuthology.orchestra.run.smithi145.stdout:instance: client.15522 (3)192.168.0.1:0/1614148170 2024-03-28T18:38:10.801 INFO:teuthology.orchestra.run.smithi145.stdout:blocklisted: false 2024-03-28T18:38:10.802 INFO:teuthology.orchestra.run.smithi145.stderr:0+1 records in 2024-03-28T18:38:10.802 INFO:teuthology.orchestra.run.smithi145.stderr:0+1 records out 2024-03-28T18:38:10.802 INFO:teuthology.orchestra.run.smithi145.stderr:70 bytes copied, 3.0156e-05 s, 2.3 MB/s 2024-03-28T18:38:10.803 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T18:38:10.814 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:38:10.814 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:38:10.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 -- 172.21.15.145:0/1523825269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 msgr2=0x7f016c109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 --2- 172.21.15.145:0/1523825269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c109830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0164009ca0 tx=0x7f016402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 -- 172.21.15.145:0/1523825269 shutdown_connections 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 --2- 172.21.15.145:0/1523825269 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c109830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 --2- 172.21.15.145:0/1523825269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c104b10 0x7f016c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 --2- 172.21.15.145:0/1523825269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c1021e0 0x7f016c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.951 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 -- 172.21.15.145:0/1523825269 >> 172.21.15.145:0/1523825269 conn(0x7f016c0fbb30 msgr2=0x7f016c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 -- 172.21.15.145:0/1523825269 shutdown_connections 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.948+0000 7f0172027640 1 -- 172.21.15.145:0/1523825269 wait complete. 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 Processor -- start 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 -- start start 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 0x7f016c197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 0x7f016c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c10d340 con 0x7f016c107440 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c10d1c0 con 0x7f016c104b10 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f0172027640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c10d4c0 con 0x7f016c1021e0 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f016bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f016b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 0x7f016c197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:10.952 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f016bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56910/0 (socket says 172.21.15.145:56910) 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f016bfff640 1 -- 172.21.15.145:0/1660442242 learned_addr learned my addr 172.21.15.145:0/1660442242 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.949+0000 7f016affd640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 0x7f016c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 -- 172.21.15.145:0/1660442242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 msgr2=0x7f016c197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 0x7f016c197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 -- 172.21.15.145:0/1660442242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 msgr2=0x7f016c197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 0x7f016c197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 -- 172.21.15.145:0/1660442242 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c19f520 con 0x7f016c107440 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016b7fe640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 0x7f016c197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016affd640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 0x7f016c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f016bfff640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0164037040 tx=0x7f0164032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0164041020 con 0x7f016c107440 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0164034cd0 con 0x7f016c107440 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c19f7b0 con 0x7f016c107440 2024-03-28T18:38:10.953 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0164039040 con 0x7f016c107440 2024-03-28T18:38:10.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.950+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c19fca0 con 0x7f016c107440 2024-03-28T18:38:10.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.951+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c0fd620 con 0x7f016c107440 2024-03-28T18:38:10.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.953+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0164044020 con 0x7f016c107440 2024-03-28T18:38:10.954 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.953+0000 7f0168ff9640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 0x7f01540a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:10.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.953+0000 7f016b7fe640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 0x7f01540a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:10.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.953+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f01640f0cb0 con 0x7f016c107440 2024-03-28T18:38:10.955 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.954+0000 7f016b7fe640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 0x7f01540a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f015c0095f0 tx=0x7f015c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:10.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:10.958+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f016404b020 con 0x7f016c107440 2024-03-28T18:38:10.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:10 smithi164 ceph-mon[36248]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:38:11.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:10 smithi145 ceph-mon[32151]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:38:11.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:10 smithi192 ceph-mon[36589]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:38:11.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.182+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f016c0622a0 con 0x7f016c107440 2024-03-28T18:38:11.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.685+0000 7f0168ff9640 1 -- 172.21.15.145:0/1660442242 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f01640bda30 con 0x7f016c107440 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 msgr2=0x7f01540a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 0x7f01540a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f015c0095f0 tx=0x7f015c005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 msgr2=0x7f016c19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0164037040 tx=0x7f0164032f40 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 shutdown_connections 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016c107440 0x7f016c19eeb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f016c104b10 0x7f016c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f01540a28a0 0x7f01540a4d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.687+0000 7f0172027640 1 --2- 172.21.15.145:0/1660442242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f016c1021e0 0x7f016c197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.688+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 >> 172.21.15.145:0/1660442242 conn(0x7f016c0fbb30 msgr2=0x7f016c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.688+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 shutdown_connections 2024-03-28T18:38:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.688+0000 7f0172027640 1 -- 172.21.15.145:0/1660442242 wait complete. 2024-03-28T18:38:11.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-28T18:38:11.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 -- 172.21.15.145:0/145090592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 msgr2=0x7fc99c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:11.805 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 --2- 172.21.15.145:0/145090592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c101c30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc990009900 tx=0x7fc99002bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 -- 172.21.15.145:0/145090592 shutdown_connections 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 --2- 172.21.15.145:0/145090592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c102170 0x7fc99c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 --2- 172.21.15.145:0/145090592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c101c30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 --2- 172.21.15.145:0/145090592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c1005c0 0x7fc99c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 -- 172.21.15.145:0/145090592 >> 172.21.15.145:0/145090592 conn(0x7fc99c0fbb30 msgr2=0x7fc99c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 -- 172.21.15.145:0/145090592 shutdown_connections 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.805+0000 7fc9a2875640 1 -- 172.21.15.145:0/145090592 wait complete. 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 Processor -- start 2024-03-28T18:38:11.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 -- start start 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c1005c0 0x7fc99c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 0x7fc99c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c10c280 con 0x7fc99c1017b0 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c10c100 con 0x7fc99c1005c0 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a2875640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c10c400 con 0x7fc99c102170 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc99b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc99b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56914/0 (socket says 172.21.15.145:56914) 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc99b7fe640 1 -- 172.21.15.145:0/3635172110 learned_addr learned my addr 172.21.15.145:0/3635172110 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc99bfff640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c1005c0 0x7fc99c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:11.808 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.806+0000 7fc9a0e12640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 0x7fc99c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 -- 172.21.15.145:0/3635172110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 msgr2=0x7fc99c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 0x7fc99c1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 -- 172.21.15.145:0/3635172110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c1005c0 msgr2=0x7fc99c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c1005c0 0x7fc99c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 -- 172.21.15.145:0/3635172110 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c1a17b0 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9a0e12640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 0x7fc99c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc99b7fe640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc990000c00 tx=0x7fc99002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990043020 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9900054e0 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c1a1a40 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990003bd0 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.807+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c1a1f90 con 0x7fc99c1017b0 2024-03-28T18:38:11.809 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.808+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c1a2160 con 0x7fc99c1017b0 2024-03-28T18:38:11.810 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.809+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc990046020 con 0x7fc99c1017b0 2024-03-28T18:38:11.810 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.810+0000 7fc9997fa640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 0x7fc9840a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:11.810 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.810+0000 7fc99bfff640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 0x7fc9840a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:11.810 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.810+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc9900f0e60 con 0x7fc99c1017b0 2024-03-28T18:38:11.810 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.810+0000 7fc99bfff640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 0x7fc9840a4d60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc99c101610 tx=0x7fc98c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:11.816 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:11.816+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc99004d020 con 0x7fc99c1017b0 2024-03-28T18:38:11.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:11 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T18:38:12.022 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.022+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 --> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7fc99c0fdc80 con 0x7fc9840a28a0 2024-03-28T18:38:12.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:11 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T18:38:12.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.038+0000 7fc9997fa640 1 -- 172.21.15.145:0/3635172110 <== mgr.14152 v2:172.21.15.145:6800/82375532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7fc99c0fdc80 con 0x7fc9840a28a0 2024-03-28T18:38:12.037 INFO:teuthology.orchestra.run.smithi145.stdout:{} 2024-03-28T18:38:12.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 msgr2=0x7fc9840a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:12.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 0x7fc9840a4d60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc99c101610 tx=0x7fc98c005f90 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 msgr2=0x7fc99c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc990000c00 tx=0x7fc99002c650 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 shutdown_connections 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc9840a28a0 0x7fc9840a4d60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc99c102170 0x7fc99c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.040+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc99c1017b0 0x7fc99c19a0d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.041+0000 7fc9a2875640 1 --2- 172.21.15.145:0/3635172110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc99c1005c0 0x7fc99c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.041+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 >> 172.21.15.145:0/3635172110 conn(0x7fc99c0fbb30 msgr2=0x7fc99c0fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.041+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 shutdown_connections 2024-03-28T18:38:12.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:38:12.041+0000 7fc9a2875640 1 -- 172.21.15.145:0/3635172110 wait complete. 2024-03-28T18:38:12.055 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T18:38:12.065 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T18:38:12.081 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:38:12.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T18:38:12.083 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T18:38:12.099 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:38:12.100 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T18:38:12.100 INFO:tasks.workunit:timeout=3h 2024-03-28T18:38:12.100 INFO:tasks.workunit:cleanup=True 2024-03-28T18:38:12.100 DEBUG:teuthology.orchestra.run.smithi145:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:12.115 INFO:teuthology.orchestra.run.smithi145.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:12.115 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T18:38:12.115 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T18:38:12.115 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:38:12.116 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T18:38:12.116 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-28 18:36:13.108182631 +0000 2024-03-28T18:38:12.116 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-28 18:36:13.108182631 +0000 2024-03-28T18:38:12.116 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-28 18:38:10.504052498 +0000 2024-03-28T18:38:12.116 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-28 18:36:13.108182631 +0000 2024-03-28T18:38:12.116 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T18:38:12.116 DEBUG:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T18:38:12.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:11 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T18:38:12.194 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:38:12.249 INFO:tasks.workunit.client.0.smithi145.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T18:38:12.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:38:12.027+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:38:12.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:38:12.027+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:38:12.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:38:12.027+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:38:12.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:38:12.027+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:38:12.526 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:38:12.027+0000 7f0f2bf6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:38:12.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-mon[32151]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:38:12.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T18:38:12.947 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:12 smithi145 ceph-mon[32151]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:38:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:12 smithi164 ceph-mon[36248]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:38:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:12 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T18:38:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:12 smithi164 ceph-mon[36248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:38:13.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:12 smithi192 ceph-mon[36589]: pgmap v364: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:38:13.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:12 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1660442242' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T18:38:13.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:12 smithi192 ceph-mon[36589]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:38:13.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:13 smithi164 ceph-mon[36248]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:14.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:13 smithi145 ceph-mon[32151]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:13 smithi192 ceph-mon[36589]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:14 smithi164 ceph-mon[36248]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-28T18:38:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:14 smithi164 ceph-mon[36248]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T18:38:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:14 smithi145 ceph-mon[32151]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-28T18:38:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:14 smithi145 ceph-mon[32151]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T18:38:15.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:14 smithi192 ceph-mon[36589]: pgmap v365: 129 pgs: 129 active+clean; 648 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-28T18:38:15.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:14 smithi192 ceph-mon[36589]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T18:38:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:16 smithi164 ceph-mon[36248]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:17.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:16 smithi145 ceph-mon[32151]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:17.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:16 smithi192 ceph-mon[36589]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:18.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:18 smithi164 ceph-mon[36248]: pgmap v367: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:18 smithi145 ceph-mon[32151]: pgmap v367: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:19.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:18 smithi192 ceph-mon[36589]: pgmap v367: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:20.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:20 smithi164 ceph-mon[36248]: pgmap v368: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:21.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:20 smithi145 ceph-mon[32151]: pgmap v368: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:21.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:20 smithi192 ceph-mon[36589]: pgmap v368: 129 pgs: 129 active+clean; 656 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-28T18:38:22.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:22 smithi164 ceph-mon[36248]: pgmap v369: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-28T18:38:23.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:22 smithi145 ceph-mon[32151]: pgmap v369: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-28T18:38:23.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:22 smithi192 ceph-mon[36589]: pgmap v369: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.3 KiB/s wr, 13 op/s 2024-03-28T18:38:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:24 smithi164 ceph-mon[36248]: pgmap v370: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T18:38:24.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:24 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:25.002 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:24 smithi145 ceph-mon[32151]: pgmap v370: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T18:38:25.002 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:24 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:25.017 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:24 smithi192 ceph-mon[36589]: pgmap v370: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T18:38:25.017 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:24 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- 172.21.15.145:0/3916166816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 msgr2=0x7f0c78104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3916166816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78104530 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009900 tx=0x7f0c7402bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- 172.21.15.145:0/3916166816 shutdown_connections 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3916166816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 0x7f0c781097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3916166816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 0x7f0c78106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3916166816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78104530 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- 172.21.15.145:0/3916166816 >> 172.21.15.145:0/3916166816 conn(0x7f0c780fbb30 msgr2=0x7f0c780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- 172.21.15.145:0/3916166816 shutdown_connections 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- 172.21.15.145:0/3916166816 wait complete. 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 Processor -- start 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.231+0000 7f0c8073b640 1 -- start start 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 0x7f0c781000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 0x7f0c78100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810d3c0 con 0x7f0c78102140 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810d240 con 0x7f0c781073a0 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810d540 con 0x7f0c78104a70 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49638/0 (socket says 172.21.15.145:49638) 2024-03-28T18:38:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 -- 172.21.15.145:0/3759720139 learned_addr learned my addr 172.21.15.145:0/3759720139 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 0x7f0c781000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 0x7f0c78100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 -- 172.21.15.145:0/3759720139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 msgr2=0x7f0c781000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 0x7f0c781000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 -- 172.21.15.145:0/3759720139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 msgr2=0x7f0c78100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 0x7f0c78100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 -- 172.21.15.145:0/3759720139 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c78100ec0 con 0x7f0c78102140 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000c00 tx=0x7f0c7402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c6f7fe640 1 -- 172.21.15.145:0/3759720139 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74043020 con 0x7f0c78102140 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c6f7fe640 1 -- 172.21.15.145:0/3759720139 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c740054e0 con 0x7f0c78102140 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c781a1aa0 con 0x7f0c78102140 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c6f7fe640 1 -- 172.21.15.145:0/3759720139 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74003c40 con 0x7f0c78102140 2024-03-28T18:38:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.232+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c781a1ff0 con 0x7f0c78102140 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c6f7fe640 1 -- 172.21.15.145:0/3759720139 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0c74046020 con 0x7f0c78102140 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 Processor -- start 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 -- start start 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b6ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b72910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034acfb10 con 0x563034b6b310 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034a78900 con 0x563034b70480 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c83407740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034acbbb0 con 0x563034b6a920 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7e4d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7e4d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49644/0 (socket says 172.21.15.145:49644) 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7e4d7640 1 -- 172.21.15.145:0/149396354 learned_addr learned my addr 172.21.15.145:0/149396354 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b6ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b72910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/149396354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 msgr2=0x563034b72910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b72910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.235+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/149396354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 msgr2=0x563034b6fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/149396354 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630347e4210 con 0x563034b6a920 2024-03-28T18:38:25.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c6f7fe640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 0x7f0c640a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 0x7f0c640a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b6ad40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009f00 tx=0x7f0c70011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b701175ef549275e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c6f7fe640 1 -- 172.21.15.145:0/3759720139 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0c740f0f00 con 0x7f0c78102140 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c6cff9640 1 -- 172.21.15.145:0/149396354 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70004d20 con 0x563034b6a920 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c6cff9640 1 -- 172.21.15.145:0/149396354 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c70004030 con 0x563034b6a920 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.236+0000 7f0c6cff9640 1 -- 172.21.15.145:0/149396354 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70005260 con 0x563034b6a920 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c7e4d7640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 0x7f0c640a4e00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c74006010 tx=0x7f0c740dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- 172.21.15.145:0/149396354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 msgr2=0x563034b6ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b6ad40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009f00 tx=0x7f0c70011810 comp rx=0 tx=0).stop 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- 172.21.15.145:0/149396354 shutdown_connections 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b72910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b6fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 --2- 172.21.15.145:0/149396354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b6ad40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- 172.21.15.145:0/149396354 >> 172.21.15.145:0/149396354 conn(0x563034b65e90 msgr2=0x563034b66290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- 172.21.15.145:0/149396354 shutdown_connections 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- 172.21.15.145:0/149396354 wait complete. 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.237+0000 7f0c83407740 1 -- start start 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 Processor -- start 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b74c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b7c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034b73430 con 0x563034b6b310 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034a8bd50 con 0x563034b70480 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563034b735b0 con 0x563034b6a920 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c83407740 10 client.0 _ll_register_callbacks cb 0x563034a8b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:25.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49654/0 (socket says 172.21.15.145:49654) 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 -- 172.21.15.145:0/1380574339 learned_addr learned my addr 172.21.15.145:0/1380574339 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b74c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b7c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 -- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 msgr2=0x563034b74c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b74c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 -- 172.21.15.145:0/1380574339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 msgr2=0x563034b7c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b7c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 -- 172.21.15.145:0/1380574339 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630348b0620 con 0x563034b6b310 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0c68012700 tx=0x7f0c68012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e78ec75d0017c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.238+0000 7f0c61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68004020 con 0x563034b6b310 2024-03-28T18:38:25.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0c680041c0 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c70004030 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c680043c0 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 10 client.15552 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563034b7f620 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c70004030 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0c68003a40 con 0x563034b6b310 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 10 client.15552 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 20 client.15552 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034b80d40 0x563034b83130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 4 client.15552 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563034add050 con 0x563034b80d40 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.239+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034b80d40 0x563034b83130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034b80d40 0x563034b83130 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ddb744cedf5312 server_cookie=c872fbbbc81114cc in_seq=0 out_seq=0 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c61ffb640 10 client.15552 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c61ffb640 1 -- 172.21.15.145:0/1380574339 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563034add050 con 0x563034b80d40 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c61ffb640 10 client.15552 handle_command_reply: tid=0 2024-03-28T18:38:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 client.15552 shutdown 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 msgr2=0x563034b751a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0c68012700 tx=0x7f0c68012bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 shutdown_connections 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034b80d40 0x563034b83130 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b70480 0x563034b7c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6b310 0x563034b751a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c61ffb640 0 client.15552 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 --2- 172.21.15.145:0/1380574339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6a920 0x563034b74c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.240+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 >> 172.21.15.145:0/1380574339 conn(0x563034b65e90 msgr2=0x563034b65020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.241+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 shutdown_connections 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.241+0000 7f0c83407740 1 -- 172.21.15.145:0/1380574339 wait complete. 2024-03-28T18:38:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.241+0000 7f0c83407740 20 client.15552 trim_cache size 0 max 16384 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 Processor -- start 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 -- start start 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6a920 0x563034b66270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6b310 0x563034b7c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781ac2f0 con 0x563034b70480 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781adc70 con 0x563034b6a920 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c83407740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781ac950 con 0x563034b6b310 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49670/0 (socket says 172.21.15.145:49670) 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7dcd6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6b310 0x563034b7c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/3802596509 learned_addr learned my addr 172.21.15.145:0/3802596509 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6a920 0x563034b66270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.261 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/3802596509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6b310 msgr2=0x563034b7c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6b310 0x563034b7c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/3802596509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6a920 msgr2=0x563034b66270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6a920 0x563034b66270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/3802596509 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563034b6c120 con 0x563034b70480 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.257+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0c70011d10 tx=0x7f0c70011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92c66f73f8a3bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c63fff640 1 -- 172.21.15.145:0/3802596509 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7001e040 con 0x563034b70480 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c63fff640 1 -- 172.21.15.145:0/3802596509 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c70005520 con 0x563034b70480 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c63fff640 1 -- 172.21.15.145:0/3802596509 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7000fa50 con 0x563034b70480 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- 172.21.15.145:0/3802596509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 msgr2=0x563034b6c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0c70011d10 tx=0x7f0c70011810 comp rx=0 tx=0).stop 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- 172.21.15.145:0/3802596509 shutdown_connections 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b70480 0x563034b6c630 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b6b310 0x563034b7c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 --2- 172.21.15.145:0/3802596509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6a920 0x563034b66270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- 172.21.15.145:0/3802596509 >> 172.21.15.145:0/3802596509 conn(0x563034b65e90 msgr2=0x563034b60eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- 172.21.15.145:0/3802596509 shutdown_connections 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- 172.21.15.145:0/3802596509 wait complete. 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.258+0000 7f0c83407740 1 -- start start 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 Processor -- start 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6b310 0x563034b79f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b70480 0x563034b7af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781ad610 con 0x563034b6a920 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781acfb0 con 0x563034b6b310 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c781aa4b0 con 0x563034b70480 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 10 client.0 _ll_register_callbacks cb 0x7f0c68009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:49682/0 (socket says 172.21.15.145:49682) 2024-03-28T18:38:25.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/2777533237 learned_addr learned my addr 172.21.15.145:0/2777533237 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6b310 0x563034b79f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b70480 0x563034b7af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b70480 msgr2=0x563034b7af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b70480 0x563034b7af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/2777533237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6b310 msgr2=0x563034b79f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6b310 0x563034b79f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 -- 172.21.15.145:0/2777533237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563034b7b510 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c7ecd8640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006010 tx=0x7f0c70003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9125c618494c2d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70005270 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0c7000f7d0 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.259+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c70005520 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c7000f9d0 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 10 client.15558 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563034bdc6f0 con 0x563034b6a920 2024-03-28T18:38:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c70005520 con 0x563034b6a920 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0c70024040 con 0x563034b6a920 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 10 client.15558 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 20 client.15558 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034bdcdd0 0x563034bdf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 4 client.15558 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563034ad73a0 con 0x563034bdcdd0 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.260+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034bdcdd0 0x563034bdf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.261+0000 7f0c7dcd6640 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034bdcdd0 0x563034bdf1f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6e91c3d88f2413 server_cookie=68ae1c0d5ab07660 in_seq=0 out_seq=0 2024-03-28T18:38:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.261+0000 7f0c61ffb640 10 client.15558 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c61ffb640 1 -- 172.21.15.145:0/2777533237 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563034ad73a0 con 0x563034bdcdd0 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c61ffb640 10 client.15558 handle_command_reply: tid=0 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 client.15558 shutdown 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 msgr2=0x563034b79a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006010 tx=0x7f0c70003b70 comp rx=0 tx=0).stop 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 shutdown_connections 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563034bdcdd0 0x563034bdf1f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563034b70480 0x563034b7af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563034b6b310 0x563034b79f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c61ffb640 0 client.15558 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 --2- 172.21.15.145:0/2777533237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563034b6a920 0x563034b79a20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 >> 172.21.15.145:0/2777533237 conn(0x563034b65e90 msgr2=0x563034b66270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.272+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 shutdown_connections 2024-03-28T18:38:25.273 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c83407740 1 -- 172.21.15.145:0/2777533237 wait complete. 2024-03-28T18:38:25.274 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c83407740 20 client.15558 trim_cache size 0 max 16384 2024-03-28T18:38:25.274 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:38:25.274 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:38:25.274 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:38:25.274 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 msgr2=0x7f0c640a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 0x7f0c640a4e00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c74006010 tx=0x7f0c740dd040 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 msgr2=0x7f0c78101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000c00 tx=0x7f0c7402c650 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 shutdown_connections 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c781073a0 0x7f0c78100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f0c640a2940 0x7f0c640a4e00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c78104a70 0x7f0c781000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 --2- 172.21.15.145:0/3759720139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c78102140 0x7f0c78101ab0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:25.275 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 >> 172.21.15.145:0/3759720139 conn(0x7f0c780fbb30 msgr2=0x7f0c78102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:25.276 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 shutdown_connections 2024-03-28T18:38:25.276 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:25.273+0000 7f0c8073b640 1 -- 172.21.15.145:0/3759720139 wait complete. 2024-03-28T18:38:26.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:25 smithi145 ceph-mon[32151]: pgmap v371: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T18:38:26.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:25 smithi192 ceph-mon[36589]: pgmap v371: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T18:38:26.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:25 smithi164 ceph-mon[36248]: pgmap v371: 129 pgs: 129 active+clean; 658 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:27.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:26 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:26 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:26 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:38:28.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:27 smithi192 ceph-mon[36589]: pgmap v372: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:38:28.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:27 smithi164 ceph-mon[36248]: pgmap v372: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:38:28.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:27 smithi145 ceph-mon[32151]: pgmap v372: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T18:38:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:30 smithi192 ceph-mon[36589]: pgmap v373: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-28T18:38:30.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:30 smithi164 ceph-mon[36248]: pgmap v373: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-28T18:38:31.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:30 smithi145 ceph-mon[32151]: pgmap v373: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-28T18:38:32.828 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:32 smithi145 ceph-mon[32151]: pgmap v374: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:38:32.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:32 smithi192 ceph-mon[36589]: pgmap v374: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:38:32.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:32 smithi164 ceph-mon[36248]: pgmap v374: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T18:38:34.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:34 smithi192 ceph-mon[36589]: pgmap v375: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T18:38:34.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:34 smithi164 ceph-mon[36248]: pgmap v375: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T18:38:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:34 smithi145 ceph-mon[32151]: pgmap v375: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T18:38:36.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:36 smithi192 ceph-mon[36589]: pgmap v376: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T18:38:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:36 smithi164 ceph-mon[36248]: pgmap v376: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T18:38:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:36 smithi145 ceph-mon[32151]: pgmap v376: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T18:38:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 -- 172.21.15.145:0/4111104217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c0fff70 msgr2=0x7fa81c100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 --2- 172.21.15.145:0/4111104217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c0fff70 0x7fa81c100390 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009900 tx=0x7fa80c02bc80 comp rx=0 tx=0).stop 2024-03-28T18:38:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 -- 172.21.15.145:0/4111104217 shutdown_connections 2024-03-28T18:38:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 --2- 172.21.15.145:0/4111104217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 0x7fa81c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 --2- 172.21.15.145:0/4111104217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c1008d0 0x7fa81c100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 --2- 172.21.15.145:0/4111104217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c0fff70 0x7fa81c100390 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 -- 172.21.15.145:0/4111104217 >> 172.21.15.145:0/4111104217 conn(0x7fa81c0fbb30 msgr2=0x7fa81c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 -- 172.21.15.145:0/4111104217 shutdown_connections 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.363+0000 7fa822081640 1 -- 172.21.15.145:0/4111104217 wait complete. 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 Processor -- start 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 -- start start 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c0fff70 0x7fa81c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 0x7fa81c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10f530 con 0x7fa81c1008d0 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10f3b0 con 0x7fa81c0fff70 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa822081640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10f6b0 con 0x7fa81c101270 2024-03-28T18:38:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60566/0 (socket says 172.21.15.145:60566) 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81affd640 1 -- 172.21.15.145:0/2570170351 learned_addr learned my addr 172.21.15.145:0/2570170351 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c0fff70 0x7fa81c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81bfff640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 0x7fa81c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.364+0000 7fa81affd640 1 -- 172.21.15.145:0/2570170351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 msgr2=0x7fa81c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa81affd640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 0x7fa81c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa81affd640 1 -- 172.21.15.145:0/2570170351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c0fff70 msgr2=0x7fa81c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa81affd640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c0fff70 0x7fa81c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa81affd640 1 -- 172.21.15.145:0/2570170351 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1a1790 con 0x7fa81c1008d0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa81affd640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa81000ea60 tx=0x7fa81000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa818ff9640 1 -- 172.21.15.145:0/2570170351 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81000cde0 con 0x7fa81c1008d0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa818ff9640 1 -- 172.21.15.145:0/2570170351 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8100035f0 con 0x7fa81c1008d0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c1a1a80 con 0x7fa81c1008d0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa818ff9640 1 -- 172.21.15.145:0/2570170351 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810010670 con 0x7fa81c1008d0 2024-03-28T18:38:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.365+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1a1fd0 con 0x7fa81c1008d0 2024-03-28T18:38:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.367+0000 7fa824c6c740 1 Processor -- start 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.367+0000 7fa824c6c740 1 -- start start 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.367+0000 7fa818ff9640 1 -- 172.21.15.145:0/2570170351 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa8100037a0 con 0x7fa81c1008d0 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.367+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b66f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.367+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b66fee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa824c6c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b665eb10 con 0x5647b66ff480 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa824c6c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b6607900 con 0x5647b66f9920 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa824c6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b665abb0 con 0x5647b66fa310 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b66f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60568/0 (socket says 172.21.15.145:60568) 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81affd640 1 -- 172.21.15.145:0/1726659565 learned_addr learned my addr 172.21.15.145:0/1726659565 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b66fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 -- 172.21.15.145:0/1726659565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 msgr2=0x5647b66fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b66fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 -- 172.21.15.145:0/1726659565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 msgr2=0x5647b6701910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 -- 172.21.15.145:0/1726659565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647b6373210 con 0x5647b66f9920 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b66fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81affd640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa818ff9640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 0x7fa8040a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b66f9d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa814009f00 tx=0x7fa814011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f651f39d259bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 0x7fa8040a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa818ff9640 1 -- 172.21.15.145:0/2570170351 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa8100cf290 con 0x7fa81c1008d0 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa801ffb640 1 -- 172.21.15.145:0/1726659565 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814004d20 con 0x5647b66f9920 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa801ffb640 1 -- 172.21.15.145:0/1726659565 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa814004030 con 0x5647b66f9920 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.368+0000 7fa801ffb640 1 -- 172.21.15.145:0/1726659565 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814005260 con 0x5647b66f9920 2024-03-28T18:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 -- 172.21.15.145:0/1726659565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 msgr2=0x5647b66f9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b66f9d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa814009f00 tx=0x7fa814011810 comp rx=0 tx=0).stop 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 0x7fa8040a4e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa80c006010 tx=0x7fa80c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 -- 172.21.15.145:0/1726659565 shutdown_connections 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6701910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b66fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1726659565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b66f9d40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 -- 172.21.15.145:0/1726659565 >> 172.21.15.145:0/1726659565 conn(0x5647b66f4e90 msgr2=0x5647b66f5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 -- 172.21.15.145:0/1726659565 shutdown_connections 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.369+0000 7fa824c6c740 1 -- 172.21.15.145:0/1726659565 wait complete. 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 -- start start 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 Processor -- start 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6704150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b6702430 con 0x5647b66f9920 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b661ad50 con 0x5647b66ff480 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647b67025b0 con 0x5647b66fa310 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa824c6c740 10 client.0 _ll_register_callbacks cb 0x5647b661a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:53110/0 (socket says 172.21.15.145:53110) 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa81bfff640 1 -- 172.21.15.145:0/809403118 learned_addr learned my addr 172.21.15.145:0/809403118 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.370+0000 7fa81affd640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6704150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 -- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 msgr2=0x5647b6704150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6704150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 -- 172.21.15.145:0/809403118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 msgr2=0x5647b670b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 -- 172.21.15.145:0/809403118 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647b643f620 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81bfff640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703c10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009230 tx=0x7fa80c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78210d2fa34f2358 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c04e070 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa80c034440 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa814004030 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa80c05a050 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 10 client.15570 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5647b670e570 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa814004030 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa80c048550 con 0x5647b66f9920 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 10 client.15570 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.371+0000 7fa824c6c740 20 client.15570 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b670fbf0 0x5647b6711fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa824c6c740 4 client.15570 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647b666c050 con 0x5647b670fbf0 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa81affd640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b670fbf0 0x5647b6711fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa7faffd640 10 client.15570 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.372+0000 7fa81affd640 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b670fbf0 0x5647b6711fe0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6fcf65e373409a6 server_cookie=9335fbfd2d8fb0ad in_seq=0 out_seq=0 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa7faffd640 1 -- 172.21.15.145:0/809403118 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5647b666c050 con 0x5647b670fbf0 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa7faffd640 10 client.15570 handle_command_reply: tid=0 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 client.15570 shutdown 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 msgr2=0x5647b6703c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703c10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009230 tx=0x7fa80c009260 comp rx=0 tx=0).stop 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 shutdown_connections 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b670fbf0 0x5647b6711fe0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b670b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6704150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa7faffd640 0 client.15570 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 --2- 172.21.15.145:0/809403118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703c10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 >> 172.21.15.145:0/809403118 conn(0x5647b66f4e90 msgr2=0x5647b66f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 shutdown_connections 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 1 -- 172.21.15.145:0/809403118 wait complete. 2024-03-28T18:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.373+0000 7fa824c6c740 20 client.15570 trim_cache size 0 max 16384 2024-03-28T18:38:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.386+0000 7fa824c6c740 1 Processor -- start 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 -- start start 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b670b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b6703d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1ac230 con 0x5647b66f9920 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1adbb0 con 0x5647b66ff480 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa824c6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1ac890 con 0x5647b66fa310 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60592/0 (socket says 172.21.15.145:60592) 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 -- 172.21.15.145:0/1646799413 learned_addr learned my addr 172.21.15.145:0/1646799413 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81affd640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b670b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b6703d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 -- 172.21.15.145:0/1646799413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 msgr2=0x5647b670b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b670b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 -- 172.21.15.145:0/1646799413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 msgr2=0x5647b6703d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b6703d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 -- 172.21.15.145:0/1646799413 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647b6704c00 con 0x5647b66f9920 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b6703d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa80c02c1c0 tx=0x7fa80c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8bfa349580ea50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa8017fa640 1 -- 172.21.15.145:0/1646799413 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c057020 con 0x5647b66f9920 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa8017fa640 1 -- 172.21.15.145:0/1646799413 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c005510 con 0x5647b66f9920 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.387+0000 7fa8017fa640 1 -- 172.21.15.145:0/1646799413 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c04e040 con 0x5647b66f9920 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- 172.21.15.145:0/1646799413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 msgr2=0x5647b6703820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa80c02c1c0 tx=0x7fa80c009680 comp rx=0 tx=0).stop 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- 172.21.15.145:0/1646799413 shutdown_connections 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66ff480 0x5647b6703d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b670b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1646799413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66f9920 0x5647b6703820 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- 172.21.15.145:0/1646799413 >> 172.21.15.145:0/1646799413 conn(0x5647b66f4e90 msgr2=0x5647b66f5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- 172.21.15.145:0/1646799413 shutdown_connections 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- 172.21.15.145:0/1646799413 wait complete. 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 -- start start 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.388+0000 7fa824c6c740 1 Processor -- start 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b6708920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1ad550 con 0x5647b66ff480 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1acef0 con 0x5647b66f9920 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1aa3f0 con 0x5647b66fa310 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa824c6c740 10 client.0 _ll_register_callbacks cb 0x7fa80c0040f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b6708920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:35888/0 (socket says 172.21.15.145:35888) 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 -- 172.21.15.145:0/1586556723 learned_addr learned my addr 172.21.15.145:0/1586556723 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60604/0 (socket says 172.21.15.145:60604) 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 -- 172.21.15.145:0/1586556723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 msgr2=0x5647b6708920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b6708920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 -- 172.21.15.145:0/1586556723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 msgr2=0x5647b6709e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 -- 172.21.15.145:0/1586556723 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647b670a460 con 0x5647b66fa310 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b6708920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81bfff640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa81affd640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa81009b6b0 tx=0x7fa810086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e5b111d602d038 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa7faffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.389+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81009b260 con 0x5647b66fa310 2024-03-28T18:38:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa80c005510 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa8100ccae0 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa81009b080 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 10 client.25429 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5647b676b610 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa80c005510 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa810097d90 con 0x5647b66fa310 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 10 client.25429 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 10 client.25429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 20 client.25429 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b676bef0 0x5647b676e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 4 client.25429 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5647b66663a0 con 0x5647b676bef0 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.390+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b676bef0 0x5647b676e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.391+0000 7fa7faffd640 10 client.25429 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.391+0000 7fa81b7fe640 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b676bef0 0x5647b676e310 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ecdb3092a48d22 server_cookie=51aa2d63b06c63a0 in_seq=0 out_seq=0 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.395+0000 7fa7faffd640 1 -- 172.21.15.145:0/1586556723 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5647b66663a0 con 0x5647b676bef0 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.395+0000 7fa7faffd640 10 client.25429 handle_command_reply: tid=0 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.395+0000 7fa824c6c740 1 client.25429 shutdown 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.395+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 msgr2=0x5647b6708e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.395+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa81009b6b0 tx=0x7fa810086040 comp rx=0 tx=0).stop 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 shutdown_connections 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5647b676bef0 0x5647b676e310 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5647b66ff480 0x5647b6709e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa7faffd640 0 client.25429 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5647b66fa310 0x5647b6708e60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 --2- 172.21.15.145:0/1586556723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5647b66f9920 0x5647b6708920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 >> 172.21.15.145:0/1586556723 conn(0x5647b66f4e90 msgr2=0x5647b67037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 shutdown_connections 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 1 -- 172.21.15.145:0/1586556723 wait complete. 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa824c6c740 20 client.25429 trim_cache size 0 max 16384 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 msgr2=0x7fa8040a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 0x7fa8040a4e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa80c006010 tx=0x7fa80c000bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 msgr2=0x7fa81c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa81000ea60 tx=0x7fa81000ef20 comp rx=0 tx=0).stop 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 shutdown_connections 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa8040a2940 0x7fa8040a4e00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa81c101270 0x7fa81c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa81c1008d0 0x7fa81c19a0b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 --2- 172.21.15.145:0/2570170351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa81c0fff70 0x7fa81c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.396+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 >> 172.21.15.145:0/2570170351 conn(0x7fa81c0fbb30 msgr2=0x7fa81c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.397+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 shutdown_connections 2024-03-28T18:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:38.397+0000 7fa822081640 1 -- 172.21.15.145:0/2570170351 wait complete. 2024-03-28T18:38:38.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:38 smithi192 ceph-mon[36589]: pgmap v377: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T18:38:38.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:38 smithi164 ceph-mon[36248]: pgmap v377: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T18:38:39.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:38 smithi145 ceph-mon[32151]: pgmap v377: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T18:38:40.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:40 smithi192 ceph-mon[36589]: pgmap v378: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-28T18:38:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:40 smithi164 ceph-mon[36248]: pgmap v378: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-28T18:38:41.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:40 smithi145 ceph-mon[32151]: pgmap v378: 129 pgs: 129 active+clean; 660 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-28T18:38:42.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:42 smithi192 ceph-mon[36589]: pgmap v379: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:38:42.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:42 smithi164 ceph-mon[36248]: pgmap v379: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:38:43.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:42 smithi145 ceph-mon[32151]: pgmap v379: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:38:44.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:44 smithi192 ceph-mon[36589]: pgmap v380: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:38:44.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:44 smithi164 ceph-mon[36248]: pgmap v380: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:38:45.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:44 smithi145 ceph-mon[32151]: pgmap v380: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:38:46.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:46 smithi192 ceph-mon[36589]: pgmap v381: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:38:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:46 smithi164 ceph-mon[36248]: pgmap v381: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:38:47.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:46 smithi145 ceph-mon[32151]: pgmap v381: 129 pgs: 129 active+clean; 661 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:38:48.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:48 smithi192 ceph-mon[36589]: pgmap v382: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:48.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:48 smithi164 ceph-mon[36248]: pgmap v382: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:49.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:48 smithi145 ceph-mon[32151]: pgmap v382: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:50.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:50 smithi192 ceph-mon[36589]: pgmap v383: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:50 smithi164 ceph-mon[36248]: pgmap v383: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:51.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:50 smithi145 ceph-mon[32151]: pgmap v383: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.484+0000 7f51fdd9d640 1 -- 172.21.15.145:0/395810599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8102700 msgr2=0x7f51f8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.484+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/395810599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8102700 0x7f51f8102b80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009900 tx=0x7f51e802bc30 comp rx=0 tx=0).stop 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.484+0000 7f51fdd9d640 1 -- 172.21.15.145:0/395810599 shutdown_connections 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.484+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/395810599 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/395810599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8102700 0x7f51f8102b80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/395810599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8101510 0x7f51f8101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- 172.21.15.145:0/395810599 >> 172.21.15.145:0/395810599 conn(0x7f51f80fca80 msgr2=0x7f51f80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- 172.21.15.145:0/395810599 shutdown_connections 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- 172.21.15.145:0/395810599 wait complete. 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 Processor -- start 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- start start 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8102700 0x7f51f8197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f810d1a0 con 0x7f51f8101510 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f810d020 con 0x7f51f8102700 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.485+0000 7f51fdd9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f810d320 con 0x7f51f81030c0 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53092/0 (socket says 172.21.15.145:53092) 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3976118082 learned_addr learned my addr 172.21.15.145:0/3976118082 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51effff640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8102700 0x7f51f8197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3976118082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 msgr2=0x7f51f819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3976118082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8102700 msgr2=0x7f51f8197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8102700 0x7f51f8197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3976118082 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f819f3f0 con 0x7f51f8101510 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f51f400e9c0 tx=0x7f51f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51edffb640 1 -- 172.21.15.145:0/3976118082 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f400cd80 con 0x7f51f8101510 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f819f6e0 con 0x7f51f8101510 2024-03-28T18:38:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51edffb640 1 -- 172.21.15.145:0/3976118082 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51f40035f0 con 0x7f51f8101510 2024-03-28T18:38:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.486+0000 7f51edffb640 1 -- 172.21.15.145:0/3976118082 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f4010690 con 0x7f51f8101510 2024-03-28T18:38:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.487+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f819fc00 con 0x7f51f8101510 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f51edffb640 1 -- 172.21.15.145:0/3976118082 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f51f40037a0 con 0x7f51f8101510 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 Processor -- start 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 -- start start 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731daf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a7313fb10 con 0x564a731e0680 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a730e8900 con 0x564a731db510 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.489+0000 7f5201beb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a7313bbb0 con 0x564a731dab20 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53098/0 (socket says 172.21.15.145:53098) 2024-03-28T18:38:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 -- 172.21.15.145:0/3670538102 learned_addr learned my addr 172.21.15.145:0/3670538102 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731daf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51edffb640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 0x7f51e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 0x7f51e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 -- 172.21.15.145:0/3670538102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 msgr2=0x564a731daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731daf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 -- 172.21.15.145:0/3670538102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 msgr2=0x564a731e0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 -- 172.21.15.145:0/3670538102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a72e54210 con 0x564a731e0680 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51edffb640 1 -- 172.21.15.145:0/3976118082 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f51f40cf3b0 con 0x7f51f8101510 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51effff640 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009900 tx=0x7f51e8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60228478f3c59f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 0x7f51e00a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x564a731dbd70 tx=0x7f51f0004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.490+0000 7f51deffd640 1 -- 172.21.15.145:0/3670538102 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e802cb80 con 0x564a731e0680 2024-03-28T18:38:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f51deffd640 1 -- 172.21.15.145:0/3670538102 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51e8005040 con 0x564a731e0680 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f51deffd640 1 -- 172.21.15.145:0/3670538102 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80409d0 con 0x564a731e0680 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 -- 172.21.15.145:0/3670538102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 msgr2=0x564a731e2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009900 tx=0x7f51e8009560 comp rx=0 tx=0).stop 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 -- 172.21.15.145:0/3670538102 shutdown_connections 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731e2b10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 --2- 172.21.15.145:0/3670538102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731daf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 -- 172.21.15.145:0/3670538102 >> 172.21.15.145:0/3670538102 conn(0x564a731d6090 msgr2=0x564a731d6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 -- 172.21.15.145:0/3670538102 shutdown_connections 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.491+0000 7f5201beb740 1 -- 172.21.15.145:0/3670538102 wait complete. 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 -- start start 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 Processor -- start 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a731e35d0 con 0x564a731e0680 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a730fbd50 con 0x564a731db510 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a731e3750 con 0x564a731dab20 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f5201beb740 10 client.0 _ll_register_callbacks cb 0x564a730fb720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51fd59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51fcd9b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51fd59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53106/0 (socket says 172.21.15.145:53106) 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51fd59c640 1 -- 172.21.15.145:0/1888061605 learned_addr learned my addr 172.21.15.145:0/1888061605 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57258/0 (socket says 172.21.15.145:57258) 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.492+0000 7f51effff640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 -- 172.21.15.145:0/1888061605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 msgr2=0x564a731e52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 -- 172.21.15.145:0/1888061605 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 msgr2=0x564a731ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 -- 172.21.15.145:0/1888061605 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a72f20620 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fd59c640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f51f40cc950 tx=0x7f51f40d2bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d554ac37fe9ddb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f409bee0 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51e8005040 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f51f40d2cb0 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51f40cf170 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f5201beb740 10 client.25441 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564a731ef740 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.493+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51e8005040 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f51f40ccba0 con 0x564a731dab20 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 10 client.25441 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 10 client.25441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 20 client.25441 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731f0d60 0x564a731f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 4 client.25441 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a7314d050 con 0x564a731f0d60 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f51effff640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731f0d60 0x564a731f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f51effff640 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731f0d60 0x564a731f31d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a1b0b9be2c3921e server_cookie=1103fea97678ac9a in_seq=0 out_seq=0 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.494+0000 7f51d7fff640 10 client.25441 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f51d7fff640 1 -- 172.21.15.145:0/1888061605 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564a7314d050 con 0x564a731f0d60 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f51d7fff640 10 client.25441 handle_command_reply: tid=0 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 client.25441 shutdown 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 msgr2=0x564a731e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f51f40cc950 tx=0x7f51f40d2bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 shutdown_connections 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731f0d60 0x564a731f31d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731e0680 0x564a731ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f51d7fff640 0 client.25441 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731db510 0x564a731e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 --2- 172.21.15.145:0/1888061605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731dab20 0x564a731e4d90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 >> 172.21.15.145:0/1888061605 conn(0x564a731d6090 msgr2=0x564a731d5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 shutdown_connections 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 1 -- 172.21.15.145:0/1888061605 wait complete. 2024-03-28T18:38:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.495+0000 7f5201beb740 20 client.25441 trim_cache size 0 max 16384 2024-03-28T18:38:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 Processor -- start 2024-03-28T18:38:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 -- start start 2024-03-28T18:38:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 0x564a731ecd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a7cc0 con 0x564a731dab20 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a9640 con 0x564a731d6090 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f5201beb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a8320 con 0x564a731db510 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53122/0 (socket says 172.21.15.145:53122) 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 -- 172.21.15.145:0/3847479062 learned_addr learned my addr 172.21.15.145:0/3847479062 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 0x564a731ecd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 -- 172.21.15.145:0/3847479062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 msgr2=0x564a731d92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731d92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 -- 172.21.15.145:0/3847479062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 msgr2=0x564a731ecd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 0x564a731ecd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 -- 172.21.15.145:0/3847479062 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a731e3510 con 0x564a731dab20 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 0x564a731ecd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.509+0000 7f51effff640 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009900 tx=0x7f51e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc190fbe727acad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f51de7fc640 1 -- 172.21.15.145:0/3847479062 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e802c980 con 0x564a731dab20 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f51de7fc640 1 -- 172.21.15.145:0/3847479062 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51e802cca0 con 0x564a731dab20 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f51de7fc640 1 -- 172.21.15.145:0/3847479062 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8005d50 con 0x564a731dab20 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 -- 172.21.15.145:0/3847479062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 msgr2=0x564a731ec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009900 tx=0x7f51e8000bc0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 -- 172.21.15.145:0/3847479062 shutdown_connections 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731d92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731dab20 0x564a731ec2d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 --2- 172.21.15.145:0/3847479062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731d6090 0x564a731ecd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 -- 172.21.15.145:0/3847479062 >> 172.21.15.145:0/3847479062 conn(0x564a731e0180 msgr2=0x564a731d6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.510+0000 7f5201beb740 1 -- 172.21.15.145:0/3847479062 shutdown_connections 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 -- 172.21.15.145:0/3847479062 wait complete. 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 -- start start 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 Processor -- start 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 0x564a731e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731ea8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a8fe0 con 0x564a731d6090 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a8980 con 0x564a731dab20 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f81a5e80 con 0x564a731db510 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f5201beb740 10 client.0 _ll_register_callbacks cb 0x7f51f40046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f51fcd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f51fcd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53130/0 (socket says 172.21.15.145:53130) 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3656476851 learned_addr learned my addr 172.21.15.145:0/3656476851 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:38:51.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f51effff640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 0x564a731e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.511+0000 7f51fd59c640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731ea8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 msgr2=0x564a731ea8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731ea8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3656476851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 msgr2=0x564a731e98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 0x564a731e98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 -- 172.21.15.145:0/3656476851 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a731eaee0 con 0x564a731d6090 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51effff640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 0x564a731e98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51fcd9b640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f51f4097c40 tx=0x7f51f40ccdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d8b86d4305be93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f40b9060 con 0x564a731d6090 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f51f409ac80 con 0x564a731d6090 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51e802cca0 con 0x564a731d6090 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51f409bee0 con 0x564a731d6090 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f5201beb740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-03-28T18:38:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564a731ec1f0 con 0x564a731d6090 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51e802cca0 con 0x564a731d6090 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.512+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f51f4097530 con 0x564a731d6090 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 20 client.15594 populate_metadata read hostname 'smithi145' 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731e0680 0x564a7324ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 4 client.15594 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564a731473a0 con 0x564a731e0680 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f51effff640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731e0680 0x564a7324ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f51d7fff640 10 client.15594 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.513+0000 7f51effff640 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731e0680 0x564a7324ea50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a43fc91c1e5c2bc3 server_cookie=f3de7fc259b54202 in_seq=0 out_seq=0 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.519+0000 7f51d7fff640 1 -- 172.21.15.145:0/3656476851 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564a731473a0 con 0x564a731e0680 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.519+0000 7f51d7fff640 10 client.15594 handle_command_reply: tid=0 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.519+0000 7f5201beb740 1 client.15594 shutdown 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 msgr2=0x564a731e93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f51f4097c40 tx=0x7f51f40ccdb0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 shutdown_connections 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564a731e0680 0x564a7324ea50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564a731db510 0x564a731ea8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f51d7fff640 0 client.15594 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:38:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a731dab20 0x564a731e98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 --2- 172.21.15.145:0/3656476851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564a731d6090 0x564a731e93a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 >> 172.21.15.145:0/3656476851 conn(0x564a731e0180 msgr2=0x564a731df9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 shutdown_connections 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 1 -- 172.21.15.145:0/3656476851 wait complete. 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.520+0000 7f5201beb740 20 client.15594 trim_cache size 0 max 16384 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:38:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 msgr2=0x7f51e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 0x7f51e00a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x564a731dbd70 tx=0x7f51f0004ab0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 msgr2=0x7f51f81977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f51f400e9c0 tx=0x7f51f400ee80 comp rx=0 tx=0).stop 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 shutdown_connections 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f51e00a2880 0x7f51e00a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f81030c0 0x7f51f819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51f8102700 0x7f51f8197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 --2- 172.21.15.145:0/3976118082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f8101510 0x7f51f81977d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 >> 172.21.15.145:0/3976118082 conn(0x7f51f80fca80 msgr2=0x7f51f8105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 shutdown_connections 2024-03-28T18:38:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:38:51.521+0000 7f51fdd9d640 1 -- 172.21.15.145:0/3976118082 wait complete. 2024-03-28T18:38:52.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:52 smithi192 ceph-mon[36589]: pgmap v384: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:52.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:52 smithi164 ceph-mon[36248]: pgmap v384: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:52 smithi145 ceph-mon[32151]: pgmap v384: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-28T18:38:54.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:54 smithi192 ceph-mon[36589]: pgmap v385: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T18:38:54.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:54 smithi164 ceph-mon[36248]: pgmap v385: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T18:38:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:54 smithi145 ceph-mon[32151]: pgmap v385: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-28T18:38:56.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:56 smithi192 ceph-mon[36589]: pgmap v386: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-28T18:38:56.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:56 smithi164 ceph-mon[36248]: pgmap v386: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-28T18:38:57.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:56 smithi145 ceph-mon[32151]: pgmap v386: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-28T18:38:58.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:38:58 smithi192 ceph-mon[36589]: pgmap v387: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T18:38:58.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:38:58 smithi164 ceph-mon[36248]: pgmap v387: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T18:38:59.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:38:58 smithi145 ceph-mon[32151]: pgmap v387: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T18:39:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:00 smithi192 ceph-mon[36589]: pgmap v388: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-28T18:39:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:00 smithi164 ceph-mon[36248]: pgmap v388: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-28T18:39:01.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:00 smithi145 ceph-mon[32151]: pgmap v388: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-28T18:39:02.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:02 smithi192 ceph-mon[36589]: pgmap v389: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:02 smithi164 ceph-mon[36248]: pgmap v389: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:03.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:02 smithi145 ceph-mon[32151]: pgmap v389: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 -- 172.21.15.145:0/2275367058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df8102950 msgr2=0x7f1df8102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/2275367058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df8102950 0x7f1df8102dd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009930 tx=0x7f1dec02bc30 comp rx=0 tx=0).stop 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 -- 172.21.15.145:0/2275367058 shutdown_connections 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/2275367058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 0x7f1df8109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/2275367058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df8102950 0x7f1df8102dd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/2275367058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df80ff980 0x7f1df80ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 -- 172.21.15.145:0/2275367058 >> 172.21.15.145:0/2275367058 conn(0x7f1df80fb2f0 msgr2=0x7f1df80fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 -- 172.21.15.145:0/2275367058 shutdown_connections 2024-03-28T18:39:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.650+0000 7f1dfcd77640 1 -- 172.21.15.145:0/2275367058 wait complete. 2024-03-28T18:39:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 Processor -- start 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 -- start start 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df8102950 0x7f1df8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 0x7f1df819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df810d340 con 0x7f1df80ff980 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df810d1c0 con 0x7f1df8103310 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1dfcd77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df810d4c0 con 0x7f1df8102950 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44430/0 (socket says 172.21.15.145:44430) 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 -- 172.21.15.145:0/893356709 learned_addr learned my addr 172.21.15.145:0/893356709 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df8102950 0x7f1df8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 0x7f1df819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 -- 172.21.15.145:0/893356709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df8102950 msgr2=0x7f1df8197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df8102950 0x7f1df8197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 -- 172.21.15.145:0/893356709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 msgr2=0x7f1df819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 0x7f1df819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.651+0000 7f1df659c640 1 -- 172.21.15.145:0/893356709 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df819f5b0 con 0x7f1df80ff980 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1df659c640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1de800d770 tx=0x7f1de800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1de77fe640 1 -- 172.21.15.145:0/893356709 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8017070 con 0x7f1df80ff980 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1de77fe640 1 -- 172.21.15.145:0/893356709 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de8009c30 con 0x7f1df80ff980 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df819f8a0 con 0x7f1df80ff980 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df819fe10 con 0x7f1df80ff980 2024-03-28T18:39:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.652+0000 7f1de77fe640 1 -- 172.21.15.145:0/893356709 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8003dd0 con 0x7f1df80ff980 2024-03-28T18:39:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.655+0000 7f1de77fe640 1 -- 172.21.15.145:0/893356709 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1de8003f70 con 0x7f1df80ff980 2024-03-28T18:39:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.655+0000 7f1dff966740 1 Processor -- start 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 -- start start 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f569c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 0x5616f569f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f55f3b10 con 0x5616f569cd10 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f559c900 con 0x5616f5698110 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1dff966740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f55efbb0 con 0x5616f5697720 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1df5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 0x5616f569f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1df6d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1df6d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40314/0 (socket says 172.21.15.145:40314) 2024-03-28T18:39:04.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1df6d9d640 1 -- 172.21.15.145:0/1304916597 learned_addr learned my addr 172.21.15.145:0/1304916597 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:04.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1de77fe640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 0x7f1de00a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.656+0000 7f1df659c640 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f569c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 -- 172.21.15.145:0/1304916597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 msgr2=0x5616f569c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f569c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 -- 172.21.15.145:0/1304916597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 msgr2=0x5616f569f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 0x5616f569f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 -- 172.21.15.145:0/1304916597 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f5308210 con 0x5616f5697720 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 0x5616f569f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 0x7f1de00a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1de77fe640 1 -- 172.21.15.145:0/893356709 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1de80ce850 con 0x7f1df80ff980 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1df0011590 tx=0x7f1df0011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b349ef72a8ea58b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1de4ff9640 1 -- 172.21.15.145:0/1304916597 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0017050 con 0x5616f5697720 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1de4ff9640 1 -- 172.21.15.145:0/1304916597 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1df000fd20 con 0x5616f5697720 2024-03-28T18:39:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.657+0000 7f1de4ff9640 1 -- 172.21.15.145:0/1304916597 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df000ea80 con 0x5616f5697720 2024-03-28T18:39:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 -- 172.21.15.145:0/1304916597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 msgr2=0x5616f5697b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 0x7f1de00a4e80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1df0022000 tx=0x7f1df0023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1df0011590 tx=0x7f1df0011a50 comp rx=0 tx=0).stop 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 -- 172.21.15.145:0/1304916597 shutdown_connections 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f569cd10 0x5616f569f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f569c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 --2- 172.21.15.145:0/1304916597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f5697720 0x5616f5697b40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.658+0000 7f1dff966740 1 -- 172.21.15.145:0/1304916597 >> 172.21.15.145:0/1304916597 conn(0x5616f5689e10 msgr2=0x5616f568a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 -- 172.21.15.145:0/1304916597 shutdown_connections 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 -- 172.21.15.145:0/1304916597 wait complete. 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 -- start start 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 Processor -- start 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 0x5616f56a1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.659+0000 7f1dff966740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f569fca0 con 0x5616f5698110 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dff966740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f55afd50 con 0x5616f5697720 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dff966740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f569fe20 con 0x5616f569cd10 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dff966740 10 client.0 _ll_register_callbacks cb 0x5616f55af720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df6d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df5d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 0x5616f56a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:45898/0 (socket says 172.21.15.145:45898) 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 -- 172.21.15.145:0/2438160203 learned_addr learned my addr 172.21.15.145:0/2438160203 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 -- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 msgr2=0x5616f56a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 -- 172.21.15.145:0/2438160203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 msgr2=0x5616f56a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 0x5616f56a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 -- 172.21.15.145:0/2438160203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616f53d4620 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 0x5616f56a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1df659c640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009610 tx=0x7f1de809a690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc74d87a44901169 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8020070 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1df000fd20 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.660+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1de80973a0 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de809edd0 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 10 client.25361 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5616f56abde0 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1df000fd20 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1de809a180 con 0x5616f5697720 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 10 client.25361 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 10 client.25361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 20 client.25361 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f56ad930 0x5616f56afd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 4 client.25361 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:39:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616f5601050 con 0x5616f56ad930 2024-03-28T18:39:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.661+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f56ad930 0x5616f56afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.662+0000 7f1dddffb640 10 client.25361 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.662+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f56ad930 0x5616f56afd70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=200b83fb27de3bb9 server_cookie=c5b181c3faba29ee in_seq=0 out_seq=0 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.663+0000 7f1dddffb640 1 -- 172.21.15.145:0/2438160203 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5616f5601050 con 0x5616f56ad930 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.663+0000 7f1dddffb640 10 client.25361 handle_command_reply: tid=0 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.663+0000 7f1dff966740 1 client.25361 shutdown 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 msgr2=0x5616f56a1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009610 tx=0x7f1de809a690 comp rx=0 tx=0).stop 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 shutdown_connections 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f56ad930 0x5616f56afd70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5698110 0x5616f56a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 --2- 172.21.15.145:0/2438160203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5697720 0x5616f56a1430 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dddffb640 0 client.25361 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 >> 172.21.15.145:0/2438160203 conn(0x5616f5689e10 msgr2=0x5616f5696db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 shutdown_connections 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 1 -- 172.21.15.145:0/2438160203 wait complete. 2024-03-28T18:39:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.664+0000 7f1dff966740 20 client.25361 trim_cache size 0 max 16384 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 Processor -- start 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 -- start start 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd800f9c0 con 0x5616f5697720 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8011340 con 0x5616f5698110 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1dff966740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8010020 con 0x5616f569cd10 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:45908/0 (socket says 172.21.15.145:45908) 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df6d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44456/0 (socket says 172.21.15.145:44456) 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 -- 172.21.15.145:0/2662261202 learned_addr learned my addr 172.21.15.145:0/2662261202 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 -- 172.21.15.145:0/2662261202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 msgr2=0x5616f56a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 -- 172.21.15.145:0/2662261202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 msgr2=0x5616f56a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.689+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.690+0000 7f1df5d9b640 1 -- 172.21.15.145:0/2662261202 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f568db00 con 0x5616f5698110 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.690+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009930 tx=0x7f1dec02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ff10ebb7661d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.690+0000 7f1ddffff640 1 -- 172.21.15.145:0/2662261202 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0042c0 con 0x5616f5698110 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.690+0000 7f1ddffff640 1 -- 172.21.15.145:0/2662261202 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1dec04b070 con 0x5616f5698110 2024-03-28T18:39:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.690+0000 7f1ddffff640 1 -- 172.21.15.145:0/2662261202 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec004cf0 con 0x5616f5698110 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- 172.21.15.145:0/2662261202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 msgr2=0x5616f56a8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009930 tx=0x7f1dec02e040 comp rx=0 tx=0).stop 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- 172.21.15.145:0/2662261202 shutdown_connections 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a8970 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- 172.21.15.145:0/2662261202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- 172.21.15.145:0/2662261202 >> 172.21.15.145:0/2662261202 conn(0x5616f5689e10 msgr2=0x5616f568a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- 172.21.15.145:0/2662261202 shutdown_connections 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- 172.21.15.145:0/2662261202 wait complete. 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- start start 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 Processor -- start 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8010ce0 con 0x5616f5697720 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8010680 con 0x5616f5698110 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd800db80 con 0x5616f569cd10 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1df659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44470/0 (socket says 172.21.15.145:44470) 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1df659c640 1 -- 172.21.15.145:0/3288803837 learned_addr learned my addr 172.21.15.145:0/3288803837 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1dff966740 10 client.0 _ll_register_callbacks cb 0x7f1de809ab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.691+0000 7f1df6d9d640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 -- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 msgr2=0x5616f56a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 -- 172.21.15.145:0/3288803837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 msgr2=0x5616f56a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 -- 172.21.15.145:0/3288803837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616f56a7a50 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df659c640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1df5d9b640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a6450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5616f567e7e0 tx=0x7f1dec02c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7995e6ae5a61bee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec053070 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1dec0035f0 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1dec04b070 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dec05c050 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dff966740 10 client.25367 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5616f5709240 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.692+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1dec04b070 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1dec056020 con 0x5616f5698110 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 10 client.25367 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 10 client.25367 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 20 client.25367 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f5709aa0 0x5616f570bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 4 client.25367 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5616f55fb3a0 con 0x5616f5709aa0 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1df659c640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f5709aa0 0x5616f570bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1dddffb640 10 client.25367 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:04.699 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.693+0000 7f1df659c640 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f5709aa0 0x5616f570bec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df90a6b5a522c86c server_cookie=ecb6ad5e67893dc3 in_seq=0 out_seq=0 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.696+0000 7f1dddffb640 1 -- 172.21.15.145:0/3288803837 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5616f55fb3a0 con 0x5616f5709aa0 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.696+0000 7f1dddffb640 10 client.25367 handle_command_reply: tid=0 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.696+0000 7f1dff966740 1 client.25367 shutdown 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 msgr2=0x5616f56a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a6450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5616f567e7e0 tx=0x7f1dec02c680 comp rx=0 tx=0).stop 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 shutdown_connections 2024-03-28T18:39:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5616f5709aa0 0x5616f570bec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5616f569cd10 0x5616f56a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dddffb640 0 client.25367 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5616f5698110 0x5616f56a6450 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 --2- 172.21.15.145:0/3288803837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616f5697720 0x5616f56a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 >> 172.21.15.145:0/3288803837 conn(0x5616f5689e10 msgr2=0x5616f56a1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 shutdown_connections 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 1 -- 172.21.15.145:0/3288803837 wait complete. 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dff966740 20 client.25367 trim_cache size 0 max 16384 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:39:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 msgr2=0x7f1de00a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 0x7f1de00a4e80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1df0022000 tx=0x7f1df0023000 comp rx=0 tx=0).stop 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 msgr2=0x7f1df8197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1de800d770 tx=0x7f1de800dc30 comp rx=0 tx=0).stop 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 shutdown_connections 2024-03-28T18:39:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df8103310 0x7f1df819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f1de00a29c0 0x7f1de00a4e80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1df8102950 0x7f1df8197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 --2- 172.21.15.145:0/893356709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1df80ff980 0x7f1df8197990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 >> 172.21.15.145:0/893356709 conn(0x7f1df80fb2f0 msgr2=0x7f1df8105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.697+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 shutdown_connections 2024-03-28T18:39:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:04.698+0000 7f1dfcd77640 1 -- 172.21.15.145:0/893356709 wait complete. 2024-03-28T18:39:04.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:04 smithi192 ceph-mon[36589]: pgmap v390: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T18:39:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:04 smithi164 ceph-mon[36248]: pgmap v390: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T18:39:05.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:04 smithi145 ceph-mon[32151]: pgmap v390: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T18:39:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:06 smithi192 ceph-mon[36589]: pgmap v391: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:06 smithi164 ceph-mon[36248]: pgmap v391: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:07.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:06 smithi145 ceph-mon[32151]: pgmap v391: 129 pgs: 129 active+clean; 671 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-28T18:39:08.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:08 smithi192 ceph-mon[36589]: pgmap v392: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:08.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:08 smithi164 ceph-mon[36248]: pgmap v392: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:09.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:08 smithi145 ceph-mon[32151]: pgmap v392: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:10.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:10 smithi192 ceph-mon[36589]: pgmap v393: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:10.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:10 smithi164 ceph-mon[36248]: pgmap v393: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:11.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:10 smithi145 ceph-mon[32151]: pgmap v393: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:12.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:12 smithi192 ceph-mon[36589]: pgmap v394: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:12 smithi164 ceph-mon[36248]: pgmap v394: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:12 smithi145 ceph-mon[32151]: pgmap v394: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:14.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:14 smithi192 ceph-mon[36589]: pgmap v395: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:14 smithi164 ceph-mon[36248]: pgmap v395: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:15.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:14 smithi145 ceph-mon[32151]: pgmap v395: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:16.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:16 smithi192 ceph-mon[36589]: pgmap v396: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:16.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:16 smithi164 ceph-mon[36248]: pgmap v396: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:17.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:16 smithi145 ceph-mon[32151]: pgmap v396: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 -- 172.21.15.145:0/1874220502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58107460 msgr2=0x7efd58109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 --2- 172.21.15.145:0/1874220502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58107460 0x7efd58109850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efd50009ca0 tx=0x7efd5002c100 comp rx=0 tx=0).stop 2024-03-28T18:39:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 -- 172.21.15.145:0/1874220502 shutdown_connections 2024-03-28T18:39:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 --2- 172.21.15.145:0/1874220502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58107460 0x7efd58109850 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 --2- 172.21.15.145:0/1874220502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58104b30 0x7efd58106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 --2- 172.21.15.145:0/1874220502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58102200 0x7efd581045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 -- 172.21.15.145:0/1874220502 >> 172.21.15.145:0/1874220502 conn(0x7efd580fbb30 msgr2=0x7efd580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 -- 172.21.15.145:0/1874220502 shutdown_connections 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 -- 172.21.15.145:0/1874220502 wait complete. 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.789+0000 7efd5ec52640 1 Processor -- start 2024-03-28T18:39:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 -- start start 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58107460 0x7efd5819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5810d370 con 0x7efd58107460 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5810d1f0 con 0x7efd58102200 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5ec52640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5810d4f0 con 0x7efd58104b30 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd57fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:35012/0 (socket says 172.21.15.145:35012) 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 -- 172.21.15.145:0/3746532159 learned_addr learned my addr 172.21.15.145:0/3746532159 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd57fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:45530/0 (socket says 172.21.15.145:45530) 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58107460 0x7efd5819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 -- 172.21.15.145:0/3746532159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 msgr2=0x7efd58197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 -- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58107460 msgr2=0x7efd5819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58107460 0x7efd5819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd5c9ee640 1 -- 172.21.15.145:0/3746532159 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5819f520 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.790+0000 7efd57fff640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009c40 tx=0x7efd4c00b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd55ffb640 1 -- 172.21.15.145:0/3746532159 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c005020 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd58100050 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd55ffb640 1 -- 172.21.15.145:0/3746532159 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd4c0056a0 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd55ffb640 1 -- 172.21.15.145:0/3746532159 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c019670 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.791+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd581005a0 con 0x7efd58102200 2024-03-28T18:39:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.794+0000 7efd55ffb640 1 -- 172.21.15.145:0/3746532159 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7efd4c0051c0 con 0x7efd58102200 2024-03-28T18:39:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.794+0000 7efd6183f740 1 Processor -- start 2024-03-28T18:39:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.794+0000 7efd6183f740 1 -- start start 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.794+0000 7efd55ffb640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 0x7efd480a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd55ffb640 1 -- 172.21.15.145:0/3746532159 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd4c0d28d0 con 0x7efd58102200 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 0x7efd480a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35326/0 (socket says 172.21.15.145:35326) 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5eae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5c9ee640 1 -- 172.21.15.145:0/4098669808 learned_addr learned my addr 172.21.15.145:0/4098669808 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5ed910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f54ab10 con 0x56042f5e5920 2024-03-28T18:39:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f4f3900 con 0x56042f5eb480 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f546bb0 con 0x56042f5e6310 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 0x7efd480a4e80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7efd5002c640 tx=0x7efd50032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd57fff640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5eae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.795+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5ed910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 -- 172.21.15.145:0/4098669808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 msgr2=0x56042f5eae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5eae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 -- 172.21.15.145:0/4098669808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 msgr2=0x56042f5e5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 -- 172.21.15.145:0/4098669808 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56042f25f210 con 0x56042f5eb480 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5ed910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd50037040 tx=0x7efd5003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efe76cf4c5aaa60a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd42ffd640 1 -- 172.21.15.145:0/4098669808 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50039040 con 0x56042f5eb480 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd42ffd640 1 -- 172.21.15.145:0/4098669808 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd50050070 con 0x56042f5eb480 2024-03-28T18:39:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.796+0000 7efd42ffd640 1 -- 172.21.15.145:0/4098669808 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5003f6a0 con 0x56042f5eb480 2024-03-28T18:39:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 msgr2=0x56042f5ed910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5ed910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd50037040 tx=0x7efd5003e000 comp rx=0 tx=0).stop 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 shutdown_connections 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5ed910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 --2- 172.21.15.145:0/4098669808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5e5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 >> 172.21.15.145:0/4098669808 conn(0x56042f5e0e90 msgr2=0x56042f5e1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 shutdown_connections 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.797+0000 7efd6183f740 1 -- 172.21.15.145:0/4098669808 wait complete. 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.798+0000 7efd6183f740 1 -- start start 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.798+0000 7efd6183f740 1 Processor -- start 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.798+0000 7efd6183f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.798+0000 7efd6183f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.798+0000 7efd6183f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f5f7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd6183f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f5ee3d0 con 0x56042f5e5920 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:35036/0 (socket says 172.21.15.145:35036) 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 -- 172.21.15.145:0/3865583104 learned_addr learned my addr 172.21.15.145:0/3865583104 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f5f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f506d50 con 0x56042f5e6310 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56042f5ee550 con 0x56042f5eb480 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd6183f740 10 client.0 _ll_register_callbacks cb 0x56042f506720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 -- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 msgr2=0x56042f5f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f5f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 -- 172.21.15.145:0/3865583104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 msgr2=0x56042f5efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56042f32b620 con 0x56042f5e6310 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd57fff640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efd440129c0 tx=0x7efd44012dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62332c051252f56b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.799+0000 7efd37fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:39:17.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4400f9e0 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd4400fb80 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd50050070 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd44025050 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd6183f740 10 client.25379 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56042f5fa560 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd4400fd80 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.800+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efd44010430 con 0x56042f5e6310 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 10 client.25379 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 10 client.25379 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 20 client.25379 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f5fc020 0x56042f5fe480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 4 client.25379 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd50050070 con 0x56042f5fc020 2024-03-28T18:39:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f5fc020 0x56042f5fe480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd37fff640 10 client.25379 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.801+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f5fc020 0x56042f5fe480 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3d93fe528827375 server_cookie=8b0d8a281fa110a in_seq=0 out_seq=0 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd37fff640 1 -- 172.21.15.145:0/3865583104 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7efd50050070 con 0x56042f5fc020 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd37fff640 10 client.25379 handle_command_reply: tid=0 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 client.25379 shutdown 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 msgr2=0x56042f5f0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efd440129c0 tx=0x7efd44012dc0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 shutdown_connections 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f5fc020 0x56042f5fe480 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f5f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f0140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 --2- 172.21.15.145:0/3865583104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 >> 172.21.15.145:0/3865583104 conn(0x56042f5e0e90 msgr2=0x56042f5e0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd37fff640 0 client.25379 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 shutdown_connections 2024-03-28T18:39:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.802+0000 7efd6183f740 1 -- 172.21.15.145:0/3865583104 wait complete. 2024-03-28T18:39:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.803+0000 7efd6183f740 20 client.25379 trim_cache size 0 max 16384 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.829+0000 7efd6183f740 1 Processor -- start 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 -- start start 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd5d1ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a7e20 con 0x56042f5e5920 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd57fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd5d1ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:35044/0 (socket says 172.21.15.145:35044) 2024-03-28T18:39:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd57fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:45568/0 (socket says 172.21.15.145:45568) 2024-03-28T18:39:17.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd5d1ef640 1 -- 172.21.15.145:0/935265221 learned_addr learned my addr 172.21.15.145:0/935265221 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:17.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a97a0 con 0x56042f5eb480 2024-03-28T18:39:17.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a8480 con 0x56042f5e6310 2024-03-28T18:39:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.830+0000 7efd5d1ef640 1 -- 172.21.15.145:0/935265221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 msgr2=0x56042f5f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd5d1ef640 1 -- 172.21.15.145:0/935265221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 msgr2=0x56042f5efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd5d1ef640 1 -- 172.21.15.145:0/935265221 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56042f5e66f0 con 0x56042f5eb480 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd50006010 tx=0x7efd50032b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2524a3047c359b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd427fc640 1 -- 172.21.15.145:0/935265221 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50058070 con 0x56042f5eb480 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd427fc640 1 -- 172.21.15.145:0/935265221 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd50050070 con 0x56042f5eb480 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.831+0000 7efd427fc640 1 -- 172.21.15.145:0/935265221 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5003f440 con 0x56042f5eb480 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 msgr2=0x56042f5f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd50006010 tx=0x7efd50032b80 comp rx=0 tx=0).stop 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 shutdown_connections 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5eb480 0x56042f5f7140 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5e6310 0x56042f5f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 --2- 172.21.15.145:0/935265221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 >> 172.21.15.145:0/935265221 conn(0x56042f5e0e90 msgr2=0x56042f5e1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 shutdown_connections 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.832+0000 7efd6183f740 1 -- 172.21.15.145:0/935265221 wait complete. 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.833+0000 7efd6183f740 1 -- start start 2024-03-28T18:39:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.833+0000 7efd6183f740 1 Processor -- start 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.833+0000 7efd6183f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.833+0000 7efd6183f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd5c9ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35364/0 (socket says 172.21.15.145:35364) 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd5c9ee640 1 -- 172.21.15.145:0/3975134102 learned_addr learned my addr 172.21.15.145:0/3975134102 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:17.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f655940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a9140 con 0x56042f5e5920 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a8ae0 con 0x56042f5e6310 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd5d1ef640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f655940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581a5fe0 con 0x56042f5eb480 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 -- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 msgr2=0x56042f655940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f655940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 -- 172.21.15.145:0/3975134102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 msgr2=0x56042f5f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.834+0000 7efd57fff640 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56042f5f4fa0 con 0x56042f5e6310 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd57fff640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f4900 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd44010820 tx=0x7efd44010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab4d596e99222258 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd37fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd6183f740 10 client.0 _ll_register_callbacks cb 0x7efd44012600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:17.837 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd37fff640 1 -- 172.21.15.145:0/3975134102 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4400fbf0 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd37fff640 1 -- 172.21.15.145:0/3975134102 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd44022040 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd50050070 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd37fff640 1 -- 172.21.15.145:0/3975134102 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd4400fd90 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd6183f740 10 client.25385 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.835+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x56042f656ca0 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd37fff640 1 -- 172.21.15.145:0/3975134102 <== mon.2 v2:172.21.15.164:3300/0 4 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efd44005110 con 0x56042f5e6310 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 10 client.25385 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 10 client.25385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 20 client.25385 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f6575a0 0x56042f659a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 4 client.25385 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56042f558050 con 0x56042f6575a0 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.836+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f6575a0 0x56042f659a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.837+0000 7efd37fff640 10 client.25385 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:17.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.837+0000 7efd5c9ee640 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f6575a0 0x56042f659a60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=451c65bc82462b7a server_cookie=328b4135dea032c1 in_seq=0 out_seq=0 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd37fff640 1 -- 172.21.15.145:0/3975134102 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56042f558050 con 0x56042f6575a0 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd37fff640 10 client.25385 handle_command_reply: tid=0 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 client.25385 shutdown 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 msgr2=0x56042f5f4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f4900 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd44010820 tx=0x7efd44010c40 comp rx=0 tx=0).stop 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 shutdown_connections 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56042f6575a0 0x56042f659a60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd37fff640 0 client.25385 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56042f5eb480 0x56042f655940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56042f5e6310 0x56042f5f4900 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 --2- 172.21.15.145:0/3975134102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56042f5e5920 0x56042f5f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.840+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 >> 172.21.15.145:0/3975134102 conn(0x56042f5e0e90 msgr2=0x56042f5e1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 shutdown_connections 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd6183f740 1 -- 172.21.15.145:0/3975134102 wait complete. 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd6183f740 20 client.25385 trim_cache size 0 max 16384 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:39:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:39:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 msgr2=0x7efd480a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 0x7efd480a4e80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7efd5002c640 tx=0x7efd50032590 comp rx=0 tx=0).stop 2024-03-28T18:39:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 msgr2=0x7efd58197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.841+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009c40 tx=0x7efd4c00b690 comp rx=0 tx=0).stop 2024-03-28T18:39:17.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 shutdown_connections 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd58107460 0x7efd5819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd480a29c0 0x7efd480a4e80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd58104b30 0x7efd58197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 --2- 172.21.15.145:0/3746532159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd58102200 0x7efd58197900 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 >> 172.21.15.145:0/3746532159 conn(0x7efd580fbb30 msgr2=0x7efd58102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 shutdown_connections 2024-03-28T18:39:17.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:17.842+0000 7efd5ec52640 1 -- 172.21.15.145:0/3746532159 wait complete. 2024-03-28T18:39:18.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:18 smithi192 ceph-mon[36589]: pgmap v397: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:18.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:18 smithi164 ceph-mon[36248]: pgmap v397: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:18 smithi145 ceph-mon[32151]: pgmap v397: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-28T18:39:20.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:20 smithi192 ceph-mon[36589]: pgmap v398: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T18:39:20.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:20 smithi164 ceph-mon[36248]: pgmap v398: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T18:39:21.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:20 smithi145 ceph-mon[32151]: pgmap v398: 129 pgs: 129 active+clean; 672 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T18:39:22.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:22 smithi192 ceph-mon[36589]: pgmap v399: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:22.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:22 smithi164 ceph-mon[36248]: pgmap v399: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:23.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:22 smithi145 ceph-mon[32151]: pgmap v399: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:39:24.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:24 smithi192 ceph-mon[36589]: pgmap v400: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:24.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:24 smithi164 ceph-mon[36248]: pgmap v400: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:25.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:24 smithi145 ceph-mon[32151]: pgmap v400: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:39:26.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:26 smithi192 ceph-mon[36589]: pgmap v401: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:39:26.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:26 smithi192 ceph-mon[36589]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:39:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:26 smithi164 ceph-mon[36248]: pgmap v401: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:39:26.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:26 smithi164 ceph-mon[36248]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:39:27.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:26 smithi145 ceph-mon[32151]: pgmap v401: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T18:39:27.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:26 smithi145 ceph-mon[32151]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:27 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:27.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:27 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:28.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:27 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:28 smithi192 ceph-mon[36589]: pgmap v402: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:28.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:28 smithi164 ceph-mon[36248]: pgmap v402: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:28.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:29.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:28 smithi145 ceph-mon[32151]: pgmap v402: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:29.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:29.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:29.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:39:30.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:30 smithi192 ceph-mon[36589]: pgmap v403: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2647502255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc102170 msgr2=0x7f5bfc108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2647502255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc102170 0x7f5bfc108a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009be0 tx=0x7f5bf402bef0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2647502255 shutdown_connections 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2647502255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc102170 0x7f5bfc108a00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2647502255 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2647502255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2647502255 >> 172.21.15.145:0/2647502255 conn(0x7f5bfc0fbb30 msgr2=0x7f5bfc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2647502255 shutdown_connections 2024-03-28T18:39:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.959+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2647502255 wait complete. 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 Processor -- start 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 -- start start 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc102170 0x7f5bfc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc10c290 con 0x7f5bfc1005c0 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc10c110 con 0x7f5bfc1017b0 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.960+0000 7f5c01eb2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc10c410 con 0x7f5bfc102170 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40066/0 (socket says 172.21.15.145:40066) 2024-03-28T18:39:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/2340098927 learned_addr learned my addr 172.21.15.145:0/2340098927 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc102170 0x7f5bfc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2340098927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc102170 msgr2=0x7f5bfc19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc102170 0x7f5bfc19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 msgr2=0x7f5bfc069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2340098927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc19f560 con 0x7f5bfc1017b0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc19c630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005fd0 tx=0x7f5bf000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bf8ff9640 1 -- 172.21.15.145:0/2340098927 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf000cd00 con 0x7f5bfc1017b0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc196460 con 0x7f5bfc1017b0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bf8ff9640 1 -- 172.21.15.145:0/2340098927 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf00137d0 con 0x7f5bfc1017b0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5bf8ff9640 1 -- 172.21.15.145:0/2340098927 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0003920 con 0x7f5bfc1017b0 2024-03-28T18:39:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.961+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc196a00 con 0x7f5bfc1017b0 2024-03-28T18:39:30.964 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:30 smithi164 ceph-mon[36248]: pgmap v403: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5bf8ff9640 1 -- 172.21.15.145:0/2340098927 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5bf0022020 con 0x7f5bfc1017b0 2024-03-28T18:39:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 Processor -- start 2024-03-28T18:39:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 -- start start 2024-03-28T18:39:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68994bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f689953910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6898b0b10 con 0x55f689951480 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f689859900 con 0x55f68994b920 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6898acbb0 con 0x55f68994c310 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5bf8ff9640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 0x7f5be40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.965+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:34238/0 (socket says 172.21.15.145:34238) 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3451365177 learned_addr learned my addr 172.21.15.145:0/3451365177 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68994bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f689953910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3451365177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 msgr2=0x55f68994bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68994bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3451365177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 msgr2=0x55f689953910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f689953910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3451365177 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6895c5210 con 0x55f68994c310 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 0x7f5be40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bf8ff9640 1 -- 172.21.15.145:0/2340098927 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5bf00d2df0 con 0x7f5bfc1017b0 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f689953910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0115f0 tx=0x7f5bec011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ed11f968bd57f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.966+0000 7f5be1ffb640 1 -- 172.21.15.145:0/3451365177 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec01e070 con 0x55f68994c310 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5be1ffb640 1 -- 172.21.15.145:0/3451365177 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bec00f660 con 0x55f68994c310 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 0x7f5be40a4de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006010 tx=0x7f5bf402bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5be1ffb640 1 -- 172.21.15.145:0/3451365177 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec003900 con 0x55f68994c310 2024-03-28T18:39:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3451365177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 msgr2=0x55f689950e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0115f0 tx=0x7f5bec011ab0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3451365177 shutdown_connections 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f689953910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f689950e10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.967+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3451365177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68994bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3451365177 >> 172.21.15.145:0/3451365177 conn(0x55f689946e90 msgr2=0x55f689947290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3451365177 shutdown_connections 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3451365177 wait complete. 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 -- start start 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 Processor -- start 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.968+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f6899561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40084/0 (socket says 172.21.15.145:40084) 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3344451507 learned_addr learned my addr 172.21.15.145:0/3344451507 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f6899561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f689951480 0x55f68995d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f689954430 con 0x55f68994b920 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f68986cd50 con 0x55f689951480 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6899545b0 con 0x55f68994c310 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f689951480 0x55f68995d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 msgr2=0x55f6899561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f6899561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3344451507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f689951480 msgr2=0x55f68995d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f689951480 0x55f68995d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f689691620 con 0x55f68994b920 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.969+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f6899561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0115c0 tx=0x7f5bec00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e8fa2a4f5e826f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5c04b7c740 10 client.0 _ll_register_callbacks cb 0x55f68986c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bec00f660 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5bdaffd640 1 -- 172.21.15.145:0/3344451507 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec01e070 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5bdaffd640 1 -- 172.21.15.145:0/3344451507 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5bec016040 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5bdaffd640 1 -- 172.21.15.145:0/3344451507 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bec009d60 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5c04b7c740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.970+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x55f689960340 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5bdaffd640 1 -- 172.21.15.145:0/3344451507 <== mon.0 v2:172.21.15.145:3300/0 4 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5bec0046c0 con 0x55f68994b920 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 20 client.15642 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f689963b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 4 client.15642 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6898be050 con 0x55f6899616f0 2024-03-28T18:39:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f689963b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5bdaffd640 10 client.15642 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.971+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f689963b60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ce19360e7b4b3f server_cookie=cf5bdca57279dd87 in_seq=0 out_seq=0 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5bdaffd640 1 -- 172.21.15.145:0/3344451507 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f6898be050 con 0x55f6899616f0 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5bdaffd640 10 client.15642 handle_command_reply: tid=0 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 client.15642 shutdown 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 msgr2=0x55f689955cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0115c0 tx=0x7f5bec00fcb0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 shutdown_connections 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f689963b60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f689951480 0x55f68995d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5bdaffd640 0 client.15642 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f6899561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/3344451507 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689955cb0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 >> 172.21.15.145:0/3344451507 conn(0x55f689946e90 msgr2=0x55f689946020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.972+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 shutdown_connections 2024-03-28T18:39:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.973+0000 7f5c04b7c740 1 -- 172.21.15.145:0/3344451507 wait complete. 2024-03-28T18:39:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.973+0000 7f5c04b7c740 20 client.15642 trim_cache size 0 max 16384 2024-03-28T18:39:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 Processor -- start 2024-03-28T18:39:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 -- start start 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689968460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f689951480 0x55f6899689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a7f30 con 0x55f68994b920 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a98b0 con 0x55f68994c310 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a8590 con 0x55f689951480 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:46122/0 (socket says 172.21.15.145:46122) 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2552891 learned_addr learned my addr 172.21.15.145:0/2552891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfbfff640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f689951480 0x55f6899689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689968460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2552891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f689951480 msgr2=0x55f6899689a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f689951480 0x55f6899689a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2552891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 msgr2=0x55f689968460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689968460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2552891 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f689942570 con 0x55f68994c310 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.987+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689968460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf009f370 tx=0x7f5bf0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccf1f4b50e30d87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5be17fa640 1 -- 172.21.15.145:0/2552891 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf00a3530 con 0x55f68994c310 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5be17fa640 1 -- 172.21.15.145:0/2552891 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf00a7b80 con 0x55f68994c310 2024-03-28T18:39:30.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5be17fa640 1 -- 172.21.15.145:0/2552891 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf00a3120 con 0x55f68994c310 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2552891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 msgr2=0x55f68995d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.988+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf009f370 tx=0x7f5bf0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2552891 shutdown_connections 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f689951480 0x55f6899689a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994c310 0x55f68995d150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2552891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f68994b920 0x55f689968460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2552891 >> 172.21.15.145:0/2552891 conn(0x55f689946e90 msgr2=0x55f689946020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2552891 shutdown_connections 2024-03-28T18:39:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2552891 wait complete. 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 -- start start 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 Processor -- start 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f68995aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.989+0000 7f5c04b7c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f68995bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a9250 con 0x55f689951480 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a8bf0 con 0x55f68994b920 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1a60f0 con 0x55f68994c310 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f68995bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f68995aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:46124/0 (socket says 172.21.15.145:46124) 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfb7fe640 1 -- 172.21.15.145:0/2681256157 learned_addr learned my addr 172.21.15.145:0/2681256157 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 10 client.0 _ll_register_callbacks cb 0x7f5bec011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2681256157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 msgr2=0x55f68995a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2681256157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 msgr2=0x55f68995bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f68995bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 -- 172.21.15.145:0/2681256157 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f68995c2a0 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bfaffd640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f68995aca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00d0590 tx=0x7f5bf0013b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78256f3962be569b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf00b8060 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bf00a7b80 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5bf0013cd0 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bf0088070 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 10 client.25498 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.990+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f6899bd030 con 0x55f68994c310 2024-03-28T18:39:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf00a7b80 con 0x55f68994c310 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5bf00d5070 con 0x55f68994c310 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 10 client.25498 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 10 client.25498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 20 client.25498 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f6899bf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 4 client.25498 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f6898b83a0 con 0x55f6899616f0 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f6899bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5bdaffd640 10 client.25498 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:30.995 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.991+0000 7f5bfb7fe640 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f6899bf840 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90e418bd9a9a7f71 server_cookie=6c7fc7bfc0287364 in_seq=0 out_seq=0 2024-03-28T18:39:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5bdaffd640 1 -- 172.21.15.145:0/2681256157 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f6898b83a0 con 0x55f6899616f0 2024-03-28T18:39:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5bdaffd640 10 client.25498 handle_command_reply: tid=0 2024-03-28T18:39:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 client.25498 shutdown 2024-03-28T18:39:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 msgr2=0x55f68995aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f68995aca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00d0590 tx=0x7f5bf0013b60 comp rx=0 tx=0).stop 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 shutdown_connections 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f6899616f0 0x55f6899bf840 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f689951480 0x55f68995bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f68994c310 0x55f68995aca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 --2- 172.21.15.145:0/2681256157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f68994b920 0x55f68995a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 >> 172.21.15.145:0/2681256157 conn(0x55f689946e90 msgr2=0x55f68994a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5bdaffd640 0 client.25498 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 shutdown_connections 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 1 -- 172.21.15.145:0/2681256157 wait complete. 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.996+0000 7f5c04b7c740 20 client.25498 trim_cache size 0 max 16384 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:39:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:39:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:39:30.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 msgr2=0x7f5be40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 0x7f5be40a4de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006010 tx=0x7f5bf402bef0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 msgr2=0x7f5bfc19c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc19c630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005fd0 tx=0x7f5bf000ee50 comp rx=0 tx=0).stop 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 shutdown_connections 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f5be40a2920 0x7f5be40a4de0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5bfc102170 0x7f5bfc19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5bfc1017b0 0x7f5bfc19c630 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.997+0000 7f5c01eb2640 1 --2- 172.21.15.145:0/2340098927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc1005c0 0x7f5bfc069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.998+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 >> 172.21.15.145:0/2340098927 conn(0x7f5bfc0fbb30 msgr2=0x7f5bfc1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.998+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 shutdown_connections 2024-03-28T18:39:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:30.998+0000 7f5c01eb2640 1 -- 172.21.15.145:0/2340098927 wait complete. 2024-03-28T18:39:31.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:30 smithi145 ceph-mon[32151]: pgmap v403: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:39:32.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:32 smithi192 ceph-mon[36589]: pgmap v404: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:39:32.961 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:32 smithi164 ceph-mon[36248]: pgmap v404: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:39:33.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:32 smithi145 ceph-mon[32151]: pgmap v404: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-28T18:39:34.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:34 smithi164 ceph-mon[36248]: pgmap v405: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:39:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:34 smithi145 ceph-mon[32151]: pgmap v405: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:39:35.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:34 smithi192 ceph-mon[36589]: pgmap v405: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:39:36.808 INFO:tasks.workunit.client.0.smithi145.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T18:39:36.808 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: git switch -c 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:Or undo this operation with: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: git switch - 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T18:39:36.813 DEBUG:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T18:39:36.880 INFO:tasks.workunit.client.0.smithi145.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T18:39:36.882 INFO:tasks.workunit.client.0.smithi145.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:39:36.882 INFO:tasks.workunit.client.0.smithi145.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T18:39:36.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:36 smithi164 ceph-mon[36248]: pgmap v406: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:39:37.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:36 smithi145 ceph-mon[32151]: pgmap v406: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:39:37.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:36 smithi192 ceph-mon[36589]: pgmap v406: 129 pgs: 129 active+clean; 673 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-28T18:39:37.636 INFO:tasks.workunit.client.0.smithi145.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T18:39:37.714 INFO:tasks.workunit.client.0.smithi145.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T18:39:37.753 INFO:tasks.workunit.client.0.smithi145.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:39:37.755 INFO:tasks.workunit.client.0.smithi145.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:39:37.755 INFO:tasks.workunit.client.0.smithi145.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T18:39:37.793 INFO:tasks.workunit.client.0.smithi145.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:39:37.797 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:39:37.797 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T18:39:37.852 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-03-28T18:39:37.853 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-03-28T18:39:37.853 DEBUG:teuthology.orchestra.run.smithi145:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-28T18:39:37.912 INFO:tasks.workunit.client.0.smithi145.stdout:getting iogen 2024-03-28T18:39:37.912 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo 'getting iogen' 2024-03-28T18:39:37.913 INFO:tasks.workunit.client.0.smithi145.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-28T18:39:37.946 INFO:tasks.workunit.client.0.smithi145.stderr:--2024-03-28 18:39:37-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-28T18:39:37.947 INFO:tasks.workunit.client.0.smithi145.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-28T18:39:37.968 INFO:tasks.workunit.client.0.smithi145.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-28T18:39:37.989 INFO:tasks.workunit.client.0.smithi145.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-28T18:39:37.989 INFO:tasks.workunit.client.0.smithi145.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-03-28T18:39:37.990 INFO:tasks.workunit.client.0.smithi145.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr: 0K ....... 100% 1.03G=0s 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr:2024-03-28 18:39:37 (1.03 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr: 2024-03-28T18:39:37.993 INFO:tasks.workunit.client.0.smithi145.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-03-28T18:39:38.009 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0 2024-03-28T18:39:38.014 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/src 2024-03-28T18:39:38.015 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/src/iogen.8 2024-03-28T18:39:38.017 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/src/iogen.c 2024-03-28T18:39:38.021 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/README 2024-03-28T18:39:38.026 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/Makefile.bsd 2024-03-28T18:39:38.029 INFO:tasks.workunit.client.0.smithi145.stdout:iogen_3.1p0/Makefile 2024-03-28T18:39:38.034 INFO:tasks.workunit.client.0.smithi145.stderr:+ cd iogen_3.1p0 2024-03-28T18:39:38.034 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo 'making iogen' 2024-03-28T18:39:38.035 INFO:tasks.workunit.client.0.smithi145.stdout:making iogen 2024-03-28T18:39:38.035 INFO:tasks.workunit.client.0.smithi145.stderr:+ make 2024-03-28T18:39:38.036 INFO:tasks.workunit.client.0.smithi145.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-03-28T18:39:38.280 INFO:tasks.workunit.client.0.smithi145.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-03-28T18:39:38.372 INFO:tasks.workunit.client.0.smithi145.stdout:running iogen 2024-03-28T18:39:38.372 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo 'running iogen' 2024-03-28T18:39:38.372 INFO:tasks.workunit.client.0.smithi145.stderr:+ ./iogen -n 5 -s 2g 2024-03-28T18:39:38.373 INFO:tasks.workunit.client.0.smithi145.stderr:child spawned 84757 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:child spawned 84758 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:child spawned 84759 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:child spawned 84760 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:child spawned 84761 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stdout:sleep for 10 min 2024-03-28T18:39:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo 'sleep for 10 min' 2024-03-28T18:39:38.375 INFO:tasks.workunit.client.0.smithi145.stderr:+ sleep 600 2024-03-28T18:39:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:38 smithi164 ceph-mon[36248]: pgmap v407: 129 pgs: 129 active+clean; 674 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:39:39.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:38 smithi145 ceph-mon[32151]: pgmap v407: 129 pgs: 129 active+clean; 674 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:39:39.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:38 smithi192 ceph-mon[36589]: pgmap v407: 129 pgs: 129 active+clean; 674 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T18:39:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:40 smithi164 ceph-mon[36248]: pgmap v408: 129 pgs: 129 active+clean; 8.7 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-03-28T18:39:41.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:40 smithi145 ceph-mon[32151]: pgmap v408: 129 pgs: 129 active+clean; 8.7 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-03-28T18:39:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:40 smithi192 ceph-mon[36589]: pgmap v408: 129 pgs: 129 active+clean; 8.7 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-03-28T18:39:42.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:42 smithi164 ceph-mon[36248]: pgmap v409: 129 pgs: 129 active+clean; 356 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 30 MiB/s wr, 79 op/s 2024-03-28T18:39:43.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:42 smithi145 ceph-mon[32151]: pgmap v409: 129 pgs: 129 active+clean; 356 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 30 MiB/s wr, 79 op/s 2024-03-28T18:39:43.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:42 smithi192 ceph-mon[36589]: pgmap v409: 129 pgs: 129 active+clean; 356 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 30 MiB/s wr, 79 op/s 2024-03-28T18:39:44.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.167+0000 7fb43d40b640 1 -- 172.21.15.145:0/2633464259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381017f0 msgr2=0x7fb438101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.167+0000 7fb43d40b640 1 --2- 172.21.15.145:0/2633464259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381017f0 0x7fb438101c70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb428009900 tx=0x7fb42802bc60 comp rx=0 tx=0).stop 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.168+0000 7fb43d40b640 1 -- 172.21.15.145:0/2633464259 shutdown_connections 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.168+0000 7fb43d40b640 1 --2- 172.21.15.145:0/2633464259 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381021b0 0x7fb438108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.168+0000 7fb43d40b640 1 --2- 172.21.15.145:0/2633464259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381017f0 0x7fb438101c70 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.168+0000 7fb43d40b640 1 --2- 172.21.15.145:0/2633464259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb438100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.168+0000 7fb43d40b640 1 -- 172.21.15.145:0/2633464259 >> 172.21.15.145:0/2633464259 conn(0x7fb4380fbb70 msgr2=0x7fb4380fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.169+0000 7fb43d40b640 1 -- 172.21.15.145:0/2633464259 shutdown_connections 2024-03-28T18:39:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.169+0000 7fb43d40b640 1 -- 172.21.15.145:0/2633464259 wait complete. 2024-03-28T18:39:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.170+0000 7fb43d40b640 1 Processor -- start 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 -- start start 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb4381977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 0x7fb438197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381021b0 0x7fb43819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43810d440 con 0x7fb4381021b0 2024-03-28T18:39:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43810d2c0 con 0x7fb4381017f0 2024-03-28T18:39:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.175+0000 7fb43d40b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43810d5c0 con 0x7fb438100600 2024-03-28T18:39:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.176+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 0x7fb438197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.176+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 0x7fb438197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:39648/0 (socket says 172.21.15.145:39648) 2024-03-28T18:39:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.176+0000 7fb4367fc640 1 -- 172.21.15.145:0/1058309113 learned_addr learned my addr 172.21.15.145:0/1058309113 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.177+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb4381977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.178+0000 7fb436ffd640 1 -- 172.21.15.145:0/1058309113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 msgr2=0x7fb438197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.178+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 0x7fb438197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.178+0000 7fb436ffd640 1 -- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381021b0 msgr2=0x7fb43819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.178+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381021b0 0x7fb43819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.178+0000 7fb436ffd640 1 -- 172.21.15.145:0/1058309113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43806b1f0 con 0x7fb438100600 2024-03-28T18:39:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.180+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb4381977d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ae10 tx=0x7fb430007610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.183+0000 7fb41ffff640 1 -- 172.21.15.145:0/1058309113 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430003990 con 0x7fb438100600 2024-03-28T18:39:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.183+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb438109460 con 0x7fb438100600 2024-03-28T18:39:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.183+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438109a00 con 0x7fb438100600 2024-03-28T18:39:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.185+0000 7fb41ffff640 1 -- 172.21.15.145:0/1058309113 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb430011c30 con 0x7fb438100600 2024-03-28T18:39:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.185+0000 7fb41ffff640 1 -- 172.21.15.145:0/1058309113 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430004350 con 0x7fb438100600 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.191+0000 7fb43fffa740 1 Processor -- start 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 -- start start 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e333080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e333650 0x55e82e338150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e29cb10 con 0x55e82e332c60 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e245900 con 0x55e82e3387c0 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.192+0000 7fb43fffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e298bb0 con 0x55e82e333650 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.193+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.193+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:39654/0 (socket says 172.21.15.145:39654) 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.193+0000 7fb4367fc640 1 -- 172.21.15.145:0/2171066642 learned_addr learned my addr 172.21.15.145:0/2171066642 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4377fe640 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e333080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb41ffff640 1 -- 172.21.15.145:0/1058309113 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb4300044f0 con 0x7fb438100600 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4367fc640 1 -- 172.21.15.145:0/2171066642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e333650 msgr2=0x55e82e338150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e333650 0x55e82e338150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4367fc640 1 -- 172.21.15.145:0/2171066642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 msgr2=0x55e82e333080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e333080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.194+0000 7fb4367fc640 1 -- 172.21.15.145:0/2171066642 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e82dfb1210 con 0x55e82e3387c0 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.195+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb42802c630 tx=0x7fb42802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f1ccc0ddaf5fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.195+0000 7fb41ffff640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 0x7fb4200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.196+0000 7fb41d7fa640 1 -- 172.21.15.145:0/2171066642 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428064030 con 0x55e82e3387c0 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.196+0000 7fb41d7fa640 1 -- 172.21.15.145:0/2171066642 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb42802cbb0 con 0x55e82e3387c0 2024-03-28T18:39:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.196+0000 7fb41d7fa640 1 -- 172.21.15.145:0/2171066642 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428005260 con 0x55e82e3387c0 2024-03-28T18:39:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.196+0000 7fb41ffff640 1 -- 172.21.15.145:0/1058309113 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fb4300cea70 con 0x7fb438100600 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 -- 172.21.15.145:0/2171066642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 msgr2=0x55e82e350de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb42802c630 tx=0x7fb42802c660 comp rx=0 tx=0).stop 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb4377fe640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 0x7fb4200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 -- 172.21.15.145:0/2171066642 shutdown_connections 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e350de0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e333650 0x55e82e338150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2171066642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e333080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.197+0000 7fb43fffa740 1 -- 172.21.15.145:0/2171066642 >> 172.21.15.145:0/2171066642 conn(0x55e82e33b6a0 msgr2=0x55e82e33baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.200+0000 7fb43fffa740 1 -- 172.21.15.145:0/2171066642 shutdown_connections 2024-03-28T18:39:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.203+0000 7fb43fffa740 1 -- 172.21.15.145:0/2171066642 wait complete. 2024-03-28T18:39:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.203+0000 7fb43fffa740 1 -- start start 2024-03-28T18:39:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.203+0000 7fb43fffa740 1 Processor -- start 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e332c60 0x55e82e341c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e349150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e33fe90 con 0x55e82e333650 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e258d50 con 0x55e82e3387c0 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82e340010 con 0x55e82e332c60 2024-03-28T18:39:44.206 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb43fffa740 10 client.0 _ll_register_callbacks cb 0x55e82e258720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.205+0000 7fb4377fe640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 0x7fb4200a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55e82e333eb0 tx=0x7fb42c009060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59500/0 (socket says 172.21.15.145:59500) 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 -- 172.21.15.145:0/2213025989 learned_addr learned my addr 172.21.15.145:0/2213025989 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb436ffd640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e332c60 0x55e82e341c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 -- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e332c60 msgr2=0x55e82e341c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e332c60 0x55e82e341c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 -- 172.21.15.145:0/2213025989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 msgr2=0x55e82e349150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e349150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.206+0000 7fb4367fc640 1 -- 172.21.15.145:0/2213025989 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e82e07d620 con 0x55e82e333650 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.207+0000 7fb4367fc640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb428009850 tx=0x7fb4280048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72535639b16b606 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.207+0000 7fb41aff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.208+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428064070 con 0x55e82e333650 2024-03-28T18:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.208+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fb428005410 con 0x55e82e333650 2024-03-28T18:39:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.208+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb42802cbb0 con 0x55e82e333650 2024-03-28T18:39:44.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.218+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb428077050 con 0x55e82e333650 2024-03-28T18:39:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.222+0000 7fb43fffa740 10 client.15657 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.222+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e82e34c4c0 con 0x55e82e333650 2024-03-28T18:39:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.223+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb42802cbb0 con 0x55e82e333650 2024-03-28T18:39:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.225+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb428060440 con 0x55e82e333650 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 10 client.15657 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 20 client.15657 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e34d810 0x55e82e34fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 4 client.15657 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:39:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.228+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e82e2aa050 con 0x55e82e34d810 2024-03-28T18:39:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.230+0000 7fb436ffd640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e34d810 0x55e82e34fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.238+0000 7fb436ffd640 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e34d810 0x55e82e34fc20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=734c211c48f4e640 server_cookie=3030327617c1acd6 in_seq=0 out_seq=0 2024-03-28T18:39:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.239+0000 7fb41aff5640 10 client.15657 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.248+0000 7fb41aff5640 1 -- 172.21.15.145:0/2213025989 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e82e2aa050 con 0x55e82e34d810 2024-03-28T18:39:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.248+0000 7fb41aff5640 10 client.15657 handle_command_reply: tid=0 2024-03-28T18:39:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.248+0000 7fb43fffa740 1 client.15657 shutdown 2024-03-28T18:39:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.248+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 msgr2=0x55e82e342180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.248+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb428009850 tx=0x7fb4280048f0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 shutdown_connections 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e34d810 0x55e82e34fc20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e3387c0 0x55e82e349150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e333650 0x55e82e342180 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 --2- 172.21.15.145:0/2213025989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e332c60 0x55e82e341c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 >> 172.21.15.145:0/2213025989 conn(0x55e82e33b6a0 msgr2=0x55e82e332020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.256+0000 7fb41aff5640 0 client.15657 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.259+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 shutdown_connections 2024-03-28T18:39:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.263+0000 7fb43fffa740 1 -- 172.21.15.145:0/2213025989 wait complete. 2024-03-28T18:39:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.265+0000 7fb43fffa740 20 client.15657 trim_cache size 0 max 16384 2024-03-28T18:39:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 Processor -- start 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 -- start start 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e334330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e349110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e334f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42400fb20 con 0x55e82e332c60 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240114a0 con 0x55e82e333650 2024-03-28T18:39:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.312+0000 7fb43fffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424010180 con 0x55e82e3387c0 2024-03-28T18:39:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.314+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e334f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.314+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e334f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:43710/0 (socket says 172.21.15.145:43710) 2024-03-28T18:39:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.314+0000 7fb4377fe640 1 -- 172.21.15.145:0/1859404201 learned_addr learned my addr 172.21.15.145:0/1859404201 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.314+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e334330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 -- 172.21.15.145:0/1859404201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 msgr2=0x55e82e334f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e334f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 -- 172.21.15.145:0/1859404201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 msgr2=0x55e82e349110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e349110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 -- 172.21.15.145:0/1859404201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e82e3340f0 con 0x55e82e332c60 2024-03-28T18:39:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.315+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e334330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4300cc130 tx=0x7fb430003db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb3692f70b37a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.316+0000 7fb41cff9640 1 -- 172.21.15.145:0/1859404201 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43009f150 con 0x55e82e332c60 2024-03-28T18:39:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.316+0000 7fb41cff9640 1 -- 172.21.15.145:0/1859404201 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4300a37a0 con 0x55e82e332c60 2024-03-28T18:39:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.317+0000 7fb43fffa740 1 -- 172.21.15.145:0/1859404201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 msgr2=0x55e82e334330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.317+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e334330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4300cc130 tx=0x7fb430003db0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.318+0000 7fb43fffa740 1 -- 172.21.15.145:0/1859404201 shutdown_connections 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.318+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e334f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.318+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e349110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.318+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1859404201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e332c60 0x55e82e334330 secure :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb4300cc130 tx=0x7fb430003db0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.318+0000 7fb43fffa740 1 -- 172.21.15.145:0/1859404201 >> 172.21.15.145:0/1859404201 conn(0x55e82e33b6a0 msgr2=0x55e82e340440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.324+0000 7fb43fffa740 1 -- 172.21.15.145:0/1859404201 shutdown_connections 2024-03-28T18:39:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.325+0000 7fb43fffa740 1 -- 172.21.15.145:0/1859404201 wait complete. 2024-03-28T18:39:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.326+0000 7fb43fffa740 1 -- start start 2024-03-28T18:39:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.326+0000 7fb43fffa740 1 Processor -- start 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e346780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e346cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424010e40 con 0x55e82e347cd0 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240107e0 con 0x55e82e333650 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42400dce0 con 0x55e82e3387c0 2024-03-28T18:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.327+0000 7fb43fffa740 10 client.0 _ll_register_callbacks cb 0x7fb4280043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.328+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.328+0000 7fb4377fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:59524/0 (socket says 172.21.15.145:59524) 2024-03-28T18:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.328+0000 7fb4377fe640 1 -- 172.21.15.145:0/1737161412 learned_addr learned my addr 172.21.15.145:0/1737161412 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.329+0000 7fb4367fc640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e346cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.329+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e346780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.331+0000 7fb4377fe640 1 -- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 msgr2=0x55e82e346cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.331+0000 7fb4377fe640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e346cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.331+0000 7fb4377fe640 1 -- 172.21.15.145:0/1737161412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 msgr2=0x55e82e346780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.331+0000 7fb4377fe640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e346780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.331+0000 7fb4377fe640 1 -- 172.21.15.145:0/1737161412 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e82e348690 con 0x55e82e347cd0 2024-03-28T18:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.335+0000 7fb4377fe640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00efb0 tx=0x7fb42c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f9b4566378d738 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.336+0000 7fb41aff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.336+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c025070 con 0x55e82e347cd0 2024-03-28T18:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.336+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fb42c013660 con 0x55e82e347cd0 2024-03-28T18:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.336+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e82e2a43a0 con 0x55e82e347cd0 2024-03-28T18:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.337+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb42c02f050 con 0x55e82e347cd0 2024-03-28T18:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.337+0000 7fb43fffa740 10 client.15663 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.337+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e82e3a9540 con 0x55e82e347cd0 2024-03-28T18:39:44.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.340+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e82e2a43a0 con 0x55e82e347cd0 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.340+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb42c02ab40 con 0x55e82e347cd0 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 10 client.15663 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 20 client.15663 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e3a9f20 0x55e82e3ac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 4 client.15663 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:39:44.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.341+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e82e222710 con 0x55e82e3a9f20 2024-03-28T18:39:44.346 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.345+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e3a9f20 0x55e82e3ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:44.348 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.347+0000 7fb436ffd640 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e3a9f20 0x55e82e3ac340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a29b31adeb4ad8a2 server_cookie=d23d06ac80119b81 in_seq=0 out_seq=0 2024-03-28T18:39:44.349 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.348+0000 7fb41aff5640 10 client.15663 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb41aff5640 1 -- 172.21.15.145:0/1737161412 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e82e222710 con 0x55e82e3a9f20 2024-03-28T18:39:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb41aff5640 10 client.15663 handle_command_reply: tid=0 2024-03-28T18:39:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 client.15663 shutdown 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 msgr2=0x55e82e348150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00efb0 tx=0x7fb42c0047c0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 shutdown_connections 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e82e3a9f20 0x55e82e3ac340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e82e347cd0 0x55e82e348150 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e82e3387c0 0x55e82e346cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 --2- 172.21.15.145:0/1737161412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e82e333650 0x55e82e346780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 >> 172.21.15.145:0/1737161412 conn(0x55e82e33b6a0 msgr2=0x55e82e340440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.501+0000 7fb41aff5640 0 client.15663 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.502+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 shutdown_connections 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.502+0000 7fb43fffa740 1 -- 172.21.15.145:0/1737161412 wait complete. 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.502+0000 7fb43fffa740 20 client.15663 trim_cache size 0 max 16384 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:39:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.503+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 msgr2=0x7fb4200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.503+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 0x7fb4200a4de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55e82e333eb0 tx=0x7fb42c009060 comp rx=0 tx=0).stop 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.503+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 msgr2=0x7fb4381977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.503+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb4381977d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ae10 tx=0x7fb430007610 comp rx=0 tx=0).stop 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 shutdown_connections 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4381021b0 0x7fb43819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4381017f0 0x7fb438197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb4200a2920 0x7fb4200a4de0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 --2- 172.21.15.145:0/1058309113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb438100600 0x7fb4381977d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 >> 172.21.15.145:0/1058309113 conn(0x7fb4380fbb70 msgr2=0x7fb438104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 shutdown_connections 2024-03-28T18:39:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:44.504+0000 7fb43d40b640 1 -- 172.21.15.145:0/1058309113 wait complete. 2024-03-28T18:39:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:44 smithi164 ceph-mon[36248]: pgmap v410: 129 pgs: 129 active+clean; 1.1 GiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 98 MiB/s wr, 94 op/s 2024-03-28T18:39:45.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:44 smithi145 ceph-mon[32151]: pgmap v410: 129 pgs: 129 active+clean; 1.1 GiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 98 MiB/s wr, 94 op/s 2024-03-28T18:39:45.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:44 smithi192 ceph-mon[36589]: pgmap v410: 129 pgs: 129 active+clean; 1.1 GiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 98 MiB/s wr, 94 op/s 2024-03-28T18:39:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:46 smithi164 ceph-mon[36248]: pgmap v411: 129 pgs: 129 active+clean; 4.5 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 388 MiB/s wr, 256 op/s 2024-03-28T18:39:47.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:46 smithi145 ceph-mon[32151]: pgmap v411: 129 pgs: 129 active+clean; 4.5 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 388 MiB/s wr, 256 op/s 2024-03-28T18:39:47.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:46 smithi192 ceph-mon[36589]: pgmap v411: 129 pgs: 129 active+clean; 4.5 GiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 388 MiB/s wr, 256 op/s 2024-03-28T18:39:49.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:48 smithi145 ceph-mon[32151]: pgmap v412: 129 pgs: 129 active+clean; 5.2 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 443 MiB/s wr, 376 op/s 2024-03-28T18:39:49.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:48 smithi192 ceph-mon[36589]: pgmap v412: 129 pgs: 129 active+clean; 5.2 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 443 MiB/s wr, 376 op/s 2024-03-28T18:39:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:48 smithi164 ceph-mon[36248]: pgmap v412: 129 pgs: 129 active+clean; 5.2 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 443 MiB/s wr, 376 op/s 2024-03-28T18:39:51.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:50 smithi145 ceph-mon[32151]: pgmap v413: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 705 MiB/s wr, 567 op/s 2024-03-28T18:39:51.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:50 smithi192 ceph-mon[36589]: pgmap v413: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 705 MiB/s wr, 567 op/s 2024-03-28T18:39:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:50 smithi164 ceph-mon[36248]: pgmap v413: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 705 MiB/s wr, 567 op/s 2024-03-28T18:39:53.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:52 smithi145 ceph-mon[32151]: pgmap v414: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 798 op/s 2024-03-28T18:39:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:52 smithi192 ceph-mon[36589]: pgmap v414: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 798 op/s 2024-03-28T18:39:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:52 smithi164 ceph-mon[36248]: pgmap v414: 129 pgs: 129 active+clean; 9.1 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 798 op/s 2024-03-28T18:39:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:54 smithi145 ceph-mon[32151]: pgmap v415: 129 pgs: 129 active+clean; 9.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 MiB/s wr, 906 op/s 2024-03-28T18:39:55.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:54 smithi192 ceph-mon[36589]: pgmap v415: 129 pgs: 129 active+clean; 9.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 MiB/s wr, 906 op/s 2024-03-28T18:39:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:54 smithi164 ceph-mon[36248]: pgmap v415: 129 pgs: 129 active+clean; 9.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 855 MiB/s wr, 906 op/s 2024-03-28T18:39:57.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:56 smithi145 ceph-mon[32151]: pgmap v416: 129 pgs: 129 active+clean; 9.9 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 983 MiB/s wr, 1.34k op/s 2024-03-28T18:39:57.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:56 smithi192 ceph-mon[36589]: pgmap v416: 129 pgs: 129 active+clean; 9.9 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 983 MiB/s wr, 1.34k op/s 2024-03-28T18:39:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:56 smithi164 ceph-mon[36248]: pgmap v416: 129 pgs: 129 active+clean; 9.9 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 983 MiB/s wr, 1.34k op/s 2024-03-28T18:39:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.717+0000 7feb4a377640 1 -- 172.21.15.145:0/1864588622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 msgr2=0x7feb3c0a88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.717+0000 7feb4a377640 1 --2- 172.21.15.145:0/1864588622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0a88d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb30009900 tx=0x7feb3002bc80 comp rx=0 tx=0).stop 2024-03-28T18:39:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.718+0000 7feb4a377640 1 -- 172.21.15.145:0/1864588622 shutdown_connections 2024-03-28T18:39:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.718+0000 7feb4a377640 1 --2- 172.21.15.145:0/1864588622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.718+0000 7feb4a377640 1 --2- 172.21.15.145:0/1864588622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a8ea0 0x7feb3c0ab2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.718+0000 7feb4a377640 1 --2- 172.21.15.145:0/1864588622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0a88d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.718+0000 7feb4a377640 1 -- 172.21.15.145:0/1864588622 >> 172.21.15.145:0/1864588622 conn(0x7feb3c0a0100 msgr2=0x7feb3c0a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:57.720 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.720+0000 7feb4a377640 1 -- 172.21.15.145:0/1864588622 shutdown_connections 2024-03-28T18:39:57.720 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.720+0000 7feb4a377640 1 -- 172.21.15.145:0/1864588622 wait complete. 2024-03-28T18:39:57.721 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.721+0000 7feb4a377640 1 Processor -- start 2024-03-28T18:39:57.721 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.721+0000 7feb4a377640 1 -- start start 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.721+0000 7feb4a377640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.721+0000 7feb4a377640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb4a377640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a8ea0 0x7feb3c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb43fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb43fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60370/0 (socket says 172.21.15.145:60370) 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb43fff640 1 -- 172.21.15.145:0/2573256138 learned_addr learned my addr 172.21.15.145:0/2573256138 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0b4d70 con 0x7feb3c0a8ea0 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0b4bf0 con 0x7feb3c0a51b0 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0b4ef0 con 0x7feb3c0a64e0 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb437fe640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb48914640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a8ea0 0x7feb3c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.722+0000 7feb437fe640 1 -- 172.21.15.145:0/2573256138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 msgr2=0x7feb3c0b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb437fe640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb437fe640 1 -- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a8ea0 msgr2=0x7feb3c0afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb437fe640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a8ea0 0x7feb3c0afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb437fe640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0b08b0 con 0x7feb3c0a64e0 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb437fe640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0b39c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0a5080 tx=0x7feb3800b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb417fa640 1 -- 172.21.15.145:0/2573256138 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb38015020 con 0x7feb3c0a64e0 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c14b7d0 con 0x7feb3c0a64e0 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.723+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c14bc30 con 0x7feb3c0a64e0 2024-03-28T18:39:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.724+0000 7feb417fa640 1 -- 172.21.15.145:0/2573256138 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb38011d10 con 0x7feb3c0a64e0 2024-03-28T18:39:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.725+0000 7feb417fa640 1 -- 172.21.15.145:0/2573256138 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800f2e0 con 0x7feb3c0a64e0 2024-03-28T18:39:57.727 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.728+0000 7feb417fa640 1 -- 172.21.15.145:0/2573256138 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7feb38018020 con 0x7feb3c0a64e0 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.729+0000 7feb417fa640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 0x7feb340a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.729+0000 7feb417fa640 1 -- 172.21.15.145:0/2573256138 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7feb380cf590 con 0x7feb3c0a64e0 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.729+0000 7feb4d03a740 1 Processor -- start 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.734+0000 7feb4d03a740 1 -- start start 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.734+0000 7feb43fff640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 0x7feb340a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301db20 0x564d9301df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 0x564d93023010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d93025b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d92f82b10 con 0x564d9301e510 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d92f2b900 con 0x564d9301db20 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb4d03a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d92f7ebb0 con 0x564d93023680 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb48914640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d93025b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb48914640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d93025b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57970/0 (socket says 172.21.15.145:57970) 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb48914640 1 -- 172.21.15.145:0/3659368030 learned_addr learned my addr 172.21.15.145:0/3659368030 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.735+0000 7feb437fe640 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 0x564d93023010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.736+0000 7feb437fe640 1 -- 172.21.15.145:0/3659368030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 msgr2=0x564d93025b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.736+0000 7feb437fe640 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d93025b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.736+0000 7feb437fe640 1 -- 172.21.15.145:0/3659368030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301db20 msgr2=0x564d9301df40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.736+0000 7feb437fe640 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301db20 0x564d9301df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.736+0000 7feb437fe640 1 -- 172.21.15.145:0/3659368030 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d92c97210 con 0x564d9301e510 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.737+0000 7feb437fe640 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 0x564d93023010 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feb3809b820 tx=0x7feb3809c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2023137cbd1cfbc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.737+0000 7feb43fff640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 0x7feb340a4d90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb30006010 tx=0x7feb30004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.739+0000 7feb2e7fc640 1 -- 172.21.15.145:0/3659368030 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380b4070 con 0x564d9301e510 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.739+0000 7feb2e7fc640 1 -- 172.21.15.145:0/3659368030 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb3809ce50 con 0x564d9301e510 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb2e7fc640 1 -- 172.21.15.145:0/3659368030 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3809af70 con 0x564d9301e510 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 -- 172.21.15.145:0/3659368030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 msgr2=0x564d93023010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 0x564d93023010 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feb3809b820 tx=0x7feb3809c220 comp rx=0 tx=0).stop 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 -- 172.21.15.145:0/3659368030 shutdown_connections 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d93025b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301e510 0x564d93023010 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 --2- 172.21.15.145:0/3659368030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301db20 0x564d9301df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 -- 172.21.15.145:0/3659368030 >> 172.21.15.145:0/3659368030 conn(0x564d93019090 msgr2=0x564d93019490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 -- 172.21.15.145:0/3659368030 shutdown_connections 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.740+0000 7feb4d03a740 1 -- 172.21.15.145:0/3659368030 wait complete. 2024-03-28T18:39:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.741+0000 7feb4d03a740 1 -- start start 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.741+0000 7feb4d03a740 1 Processor -- start 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.741+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301e510 0x564d93028340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d930266d0 con 0x564d9301db20 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d92f3ed50 con 0x564d93023680 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d93026850 con 0x564d9301e510 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb4d03a740 10 client.0 _ll_register_callbacks cb 0x564d92f3e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60376/0 (socket says 172.21.15.145:60376) 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 -- 172.21.15.145:0/1903621915 learned_addr learned my addr 172.21.15.145:0/1903621915 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb437fe640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301e510 0x564d93028340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb43fff640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 -- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301e510 msgr2=0x564d93028340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301e510 0x564d93028340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 -- 172.21.15.145:0/1903621915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 msgr2=0x564d93027e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.742+0000 7feb48914640 1 -- 172.21.15.145:0/1903621915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d92d63620 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.743+0000 7feb48914640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb44066e50 tx=0x7feb440737b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c44509090f97916a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.743+0000 7feb237fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.743+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44067e50 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.743+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7feb4407d3e0 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.743+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb3809ce50 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.755+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb4407d5e0 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.758+0000 7feb4d03a740 10 client.25418 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.758+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564d930327c0 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.758+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb3809ce50 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7feb4407c070 con 0x564d93023680 2024-03-28T18:39:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 10 client.25418 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 10 client.25418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 20 client.25418 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d93033e90 0x564d93036280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 4 client.25418 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.759+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d92f90050 con 0x564d93033e90 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.760+0000 7feb43fff640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d93033e90 0x564d93036280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.760+0000 7feb43fff640 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d93033e90 0x564d93036280 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=287f9c1981c4d8b6 server_cookie=2df7e2a0a2a05658 in_seq=0 out_seq=0 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.760+0000 7feb237fe640 10 client.25418 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.764+0000 7feb237fe640 1 -- 172.21.15.145:0/1903621915 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564d92f90050 con 0x564d93033e90 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.764+0000 7feb237fe640 10 client.25418 handle_command_reply: tid=0 2024-03-28T18:39:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.764+0000 7feb4d03a740 1 client.25418 shutdown 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 msgr2=0x564d9302f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb44066e50 tx=0x7feb440737b0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 shutdown_connections 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d93033e90 0x564d93036280 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d93023680 0x564d9302f380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301e510 0x564d93028340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 --2- 172.21.15.145:0/1903621915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.766+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 >> 172.21.15.145:0/1903621915 conn(0x564d93019090 msgr2=0x564d93018290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.767+0000 7feb237fe640 0 client.25418 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.768+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 shutdown_connections 2024-03-28T18:39:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.769+0000 7feb4d03a740 1 -- 172.21.15.145:0/1903621915 wait complete. 2024-03-28T18:39:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.771+0000 7feb4d03a740 20 client.25418 trim_cache size 0 max 16384 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.805+0000 7feb4d03a740 1 Processor -- start 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.805+0000 7feb4d03a740 1 -- start start 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.805+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.805+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb43fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb43fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57982/0 (socket says 172.21.15.145:57982) 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d93023680 0x564d93028340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb437fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb437fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60382/0 (socket says 172.21.15.145:60382) 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb43fff640 1 -- 172.21.15.145:0/4053309300 learned_addr learned my addr 172.21.15.145:0/4053309300 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2800fa20 con 0x564d93023680 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb280113a0 con 0x564d9301e510 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.806+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb28010080 con 0x564d9301db20 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.807+0000 7feb437fe640 1 -- 172.21.15.145:0/4053309300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 msgr2=0x564d93027e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.807+0000 7feb437fe640 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.807+0000 7feb437fe640 1 -- 172.21.15.145:0/4053309300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d93023680 msgr2=0x564d93028340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.807+0000 7feb437fe640 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d93023680 0x564d93028340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.807+0000 7feb437fe640 1 -- 172.21.15.145:0/4053309300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d93030800 con 0x564d9301e510 2024-03-28T18:39:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.809+0000 7feb437fe640 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3809fe30 tx=0x7feb3809b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b743e4ecb82a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.809+0000 7feb2dffb640 1 -- 172.21.15.145:0/4053309300 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380b8070 con 0x564d9301e510 2024-03-28T18:39:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.810+0000 7feb2dffb640 1 -- 172.21.15.145:0/4053309300 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb3809ce50 con 0x564d9301e510 2024-03-28T18:39:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.811+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 msgr2=0x564d9302f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.811+0000 7feb4d03a740 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3809fe30 tx=0x7feb3809b5b0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.817+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 shutdown_connections 2024-03-28T18:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.817+0000 7feb4d03a740 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d93023680 0x564d93028340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.817+0000 7feb4d03a740 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302f340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.817+0000 7feb4d03a740 1 --2- 172.21.15.145:0/4053309300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d9301db20 0x564d93027e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.817+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 >> 172.21.15.145:0/4053309300 conn(0x564d93019090 msgr2=0x564d9301f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.820+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 shutdown_connections 2024-03-28T18:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.820+0000 7feb4d03a740 1 -- 172.21.15.145:0/4053309300 wait complete. 2024-03-28T18:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.820+0000 7feb4d03a740 1 -- start start 2024-03-28T18:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 Processor -- start 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d9302e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb28010d40 con 0x564d9301db20 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb280106e0 con 0x564d9301e510 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.821+0000 7feb4d03a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2800dbe0 con 0x564d93023680 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.822+0000 7feb4d03a740 10 client.0 _ll_register_callbacks cb 0x7feb440759e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.822+0000 7feb43fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.822+0000 7feb43fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58848/0 (socket says 172.21.15.145:58848) 2024-03-28T18:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.822+0000 7feb43fff640 1 -- 172.21.15.145:0/2729378810 learned_addr learned my addr 172.21.15.145:0/2729378810 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.824+0000 7feb48914640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d9302e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.824+0000 7feb437fe640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.826+0000 7feb43fff640 1 -- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 msgr2=0x564d9302e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.826+0000 7feb43fff640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d9302e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.826+0000 7feb43fff640 1 -- 172.21.15.145:0/2729378810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 msgr2=0x564d9302d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.826+0000 7feb43fff640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.826+0000 7feb43fff640 1 -- 172.21.15.145:0/2729378810 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d9302e610 con 0x564d9301db20 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.827+0000 7feb43fff640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feb3002c290 tx=0x7feb3002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914c851f283fb88b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.827+0000 7feb237fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30057070 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb3809ce50 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7feb300046c0 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb30064050 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb4d03a740 10 client.15675 fetch_fsmap learned FSMap version 29 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.828+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564d930a07d0 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.829+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb3809ce50 con 0x564d9301db20 2024-03-28T18:39:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.829+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7feb300038a0 con 0x564d9301db20 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 10 client.15675 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 10 client.15675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 20 client.15675 populate_metadata read hostname 'smithi145' 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d930a1180 0x564d930a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 4 client.15675 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:39:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.830+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564d92f8a3a0 con 0x564d930a1180 2024-03-28T18:39:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.838+0000 7feb437fe640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d930a1180 0x564d930a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.858+0000 7feb437fe640 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d930a1180 0x564d930a35a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b229d2deea247e server_cookie=3791fd5644d73ace in_seq=0 out_seq=0 2024-03-28T18:39:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:57.858+0000 7feb237fe640 10 client.15675 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:58.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:39:57 smithi192 ceph-mon[36589]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.32k op/s 2024-03-28T18:39:58.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:39:57 smithi164 ceph-mon[36248]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.32k op/s 2024-03-28T18:39:58.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:39:57 smithi145 ceph-mon[32151]: pgmap v417: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 773 MiB/s wr, 1.32k op/s 2024-03-28T18:39:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.831+0000 7feb237fe640 1 -- 172.21.15.145:0/2729378810 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564d92f8a3a0 con 0x564d930a1180 2024-03-28T18:39:59.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.831+0000 7feb237fe640 10 client.15675 handle_command_reply: tid=0 2024-03-28T18:39:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.831+0000 7feb4d03a740 1 client.15675 shutdown 2024-03-28T18:39:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 msgr2=0x564d9302cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feb3002c290 tx=0x7feb3002e050 comp rx=0 tx=0).stop 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 shutdown_connections 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564d930a1180 0x564d930a35a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d93023680 0x564d9302e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564d9301e510 0x564d9302d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 --2- 172.21.15.145:0/2729378810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564d9301db20 0x564d9302cad0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 >> 172.21.15.145:0/2729378810 conn(0x564d93019090 msgr2=0x564d9301c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.832+0000 7feb237fe640 0 client.15675 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.833+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 shutdown_connections 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.833+0000 7feb4d03a740 1 -- 172.21.15.145:0/2729378810 wait complete. 2024-03-28T18:39:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.833+0000 7feb4d03a740 20 client.15675 trim_cache size 0 max 16384 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 msgr2=0x7feb340a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 0x7feb340a4d90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb30006010 tx=0x7feb30004f10 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 msgr2=0x7feb3c0b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0b39c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0a5080 tx=0x7feb3800b850 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 shutdown_connections 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb3c0a8ea0 0x7feb3c0afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7feb340a28d0 0x7feb340a4d90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb3c0a64e0 0x7feb3c0b39c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 --2- 172.21.15.145:0/2573256138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb3c0a51b0 0x7feb3c0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.834+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 >> 172.21.15.145:0/2573256138 conn(0x7feb3c0a0100 msgr2=0x7feb3c0a7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.836+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 shutdown_connections 2024-03-28T18:39:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:39:59.836+0000 7feb4a377640 1 -- 172.21.15.145:0/2573256138 wait complete. 2024-03-28T18:40:00.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:00 smithi192 ceph-mon[36589]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 937 MiB/s wr, 1.59k op/s 2024-03-28T18:40:00.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:00 smithi192 ceph-mon[36589]: overall HEALTH_OK 2024-03-28T18:40:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:00 smithi164 ceph-mon[36248]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 937 MiB/s wr, 1.59k op/s 2024-03-28T18:40:00.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:00 smithi164 ceph-mon[36248]: overall HEALTH_OK 2024-03-28T18:40:01.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:00 smithi145 ceph-mon[32151]: pgmap v418: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 937 MiB/s wr, 1.59k op/s 2024-03-28T18:40:01.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:00 smithi145 ceph-mon[32151]: overall HEALTH_OK 2024-03-28T18:40:02.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:02 smithi192 ceph-mon[36589]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 MiB/s wr, 1.57k op/s 2024-03-28T18:40:02.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:02 smithi164 ceph-mon[36248]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 MiB/s wr, 1.57k op/s 2024-03-28T18:40:03.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:02 smithi145 ceph-mon[32151]: pgmap v419: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 MiB/s wr, 1.57k op/s 2024-03-28T18:40:04.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:04 smithi192 ceph-mon[36589]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 683 MiB/s wr, 1.42k op/s 2024-03-28T18:40:04.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:04 smithi164 ceph-mon[36248]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 683 MiB/s wr, 1.42k op/s 2024-03-28T18:40:05.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:04 smithi145 ceph-mon[32151]: pgmap v420: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 683 MiB/s wr, 1.42k op/s 2024-03-28T18:40:06.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:06 smithi145 ceph-mon[32151]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 883 MiB/s wr, 1.66k op/s 2024-03-28T18:40:06.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:06 smithi192 ceph-mon[36589]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 883 MiB/s wr, 1.66k op/s 2024-03-28T18:40:06.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:06 smithi164 ceph-mon[36248]: pgmap v421: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 883 MiB/s wr, 1.66k op/s 2024-03-28T18:40:08.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:08 smithi164 ceph-mon[36248]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.27k op/s 2024-03-28T18:40:09.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:08 smithi145 ceph-mon[32151]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.27k op/s 2024-03-28T18:40:09.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:08 smithi192 ceph-mon[36589]: pgmap v422: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.27k op/s 2024-03-28T18:40:10.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:10 smithi164 ceph-mon[36248]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 850 MiB/s wr, 1.44k op/s 2024-03-28T18:40:11.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:10 smithi145 ceph-mon[32151]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 850 MiB/s wr, 1.44k op/s 2024-03-28T18:40:11.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:10 smithi192 ceph-mon[36589]: pgmap v423: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 850 MiB/s wr, 1.44k op/s 2024-03-28T18:40:12.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:12 smithi164 ceph-mon[36248]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.19k op/s 2024-03-28T18:40:13.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:12 smithi145 ceph-mon[32151]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.19k op/s 2024-03-28T18:40:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.116+0000 7ffaba747640 1 -- 172.21.15.145:0/2213829744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab41048e0 msgr2=0x7ffab4106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.116+0000 7ffaba747640 1 --2- 172.21.15.145:0/2213829744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab41048e0 0x7ffab4106cd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009900 tx=0x7ffaa402bc80 comp rx=0 tx=0).stop 2024-03-28T18:40:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.118+0000 7ffaba747640 1 -- 172.21.15.145:0/2213829744 shutdown_connections 2024-03-28T18:40:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.118+0000 7ffaba747640 1 --2- 172.21.15.145:0/2213829744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4107210 0x7ffab4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.118+0000 7ffaba747640 1 --2- 172.21.15.145:0/2213829744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab41048e0 0x7ffab4106cd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.118+0000 7ffaba747640 1 --2- 172.21.15.145:0/2213829744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab4101fb0 0x7ffab41043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.118+0000 7ffaba747640 1 -- 172.21.15.145:0/2213829744 >> 172.21.15.145:0/2213829744 conn(0x7ffab40fb9a0 msgr2=0x7ffab40fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.122+0000 7ffaba747640 1 -- 172.21.15.145:0/2213829744 shutdown_connections 2024-03-28T18:40:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.124+0000 7ffaba747640 1 -- 172.21.15.145:0/2213829744 wait complete. 2024-03-28T18:40:13.125 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.125+0000 7ffaba747640 1 Processor -- start 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 -- start start 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4101fb0 0x7ffab41978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab4107210 0x7ffab419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab410d270 con 0x7ffab41048e0 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab410d0f0 con 0x7ffab4107210 2024-03-28T18:40:13.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.126+0000 7ffaba747640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab410d3f0 con 0x7ffab4101fb0 2024-03-28T18:40:13.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.127+0000 7ffab8f44640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.127+0000 7ffab8f44640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53610/0 (socket says 172.21.15.145:53610) 2024-03-28T18:40:13.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.127+0000 7ffab8f44640 1 -- 172.21.15.145:0/1526497246 learned_addr learned my addr 172.21.15.145:0/1526497246 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.128+0000 7ffab9745640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4101fb0 0x7ffab41978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab4107210 0x7ffab419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab8f44640 1 -- 172.21.15.145:0/1526497246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4101fb0 msgr2=0x7ffab41978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.129 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab8f44640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4101fb0 0x7ffab41978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab8f44640 1 -- 172.21.15.145:0/1526497246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab4107210 msgr2=0x7ffab419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab8f44640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab4107210 0x7ffab419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.129+0000 7ffab8f44640 1 -- 172.21.15.145:0/1526497246 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab40ffea0 con 0x7ffab41048e0 2024-03-28T18:40:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.131+0000 7ffab8f44640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009850 tx=0x7ffaa402c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.132+0000 7ffaaa7fc640 1 -- 172.21.15.145:0/1526497246 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4043020 con 0x7ffab41048e0 2024-03-28T18:40:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.132+0000 7ffaaa7fc640 1 -- 172.21.15.145:0/1526497246 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaa40043d0 con 0x7ffab41048e0 2024-03-28T18:40:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.132+0000 7ffaaa7fc640 1 -- 172.21.15.145:0/1526497246 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4003d40 con 0x7ffab41048e0 2024-03-28T18:40:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.132+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab4100130 con 0x7ffab41048e0 2024-03-28T18:40:13.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.132+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab4100680 con 0x7ffab41048e0 2024-03-28T18:40:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.137+0000 7ffaaa7fc640 1 -- 172.21.15.145:0/1526497246 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffaa4046020 con 0x7ffab41048e0 2024-03-28T18:40:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.141+0000 7ffabe591740 1 Processor -- start 2024-03-28T18:40:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.141+0000 7ffabe591740 1 -- start start 2024-03-28T18:40:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.141+0000 7ffaaa7fc640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 0x7ffaa00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.141+0000 7ffabe591740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada35f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.141+0000 7ffabe591740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada36510 0x5596ada3b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffabe591740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffabe591740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ad99ab10 con 0x5596ada36510 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffabe591740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ad943900 con 0x5596ada35b20 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffabe591740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ad996bb0 con 0x5596ada3b680 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffaaa7fc640 1 -- 172.21.15.145:0/1526497246 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ffaa40f2280 con 0x7ffab41048e0 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffab9f46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffab9f46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:59354/0 (socket says 172.21.15.145:59354) 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffab9f46640 1 -- 172.21.15.145:0/3754794222 learned_addr learned my addr 172.21.15.145:0/3754794222 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.142+0000 7ffab8f44640 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada36510 0x5596ada3b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9745640 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada35f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 -- 172.21.15.145:0/3754794222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 msgr2=0x5596ada35f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada35f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 -- 172.21.15.145:0/3754794222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada36510 msgr2=0x5596ada3b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada36510 0x5596ada3b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 -- 172.21.15.145:0/3754794222 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ad6af210 con 0x5596ada3b680 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 0x7ffaa00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.143+0000 7ffab9f46640 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffab000a8f0 tx=0x7ffab000adb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc654d52ebf1ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.144+0000 7ffa9f7fe640 1 -- 172.21.15.145:0/3754794222 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0019070 con 0x5596ada3b680 2024-03-28T18:40:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.144+0000 7ffa9f7fe640 1 -- 172.21.15.145:0/3754794222 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffab0015920 con 0x5596ada3b680 2024-03-28T18:40:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.145+0000 7ffabe591740 1 -- 172.21.15.145:0/3754794222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 msgr2=0x5596ada3db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.145+0000 7ffabe591740 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffab000a8f0 tx=0x7ffab000adb0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.145+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 0x7ffaa00a4ec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffab001c000 tx=0x7ffab001d740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.149+0000 7ffabe591740 1 -- 172.21.15.145:0/3754794222 shutdown_connections 2024-03-28T18:40:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.149+0000 7ffabe591740 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada3db10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.149+0000 7ffabe591740 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada36510 0x5596ada3b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.149+0000 7ffabe591740 1 --2- 172.21.15.145:0/3754794222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada35f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.149+0000 7ffabe591740 1 -- 172.21.15.145:0/3754794222 >> 172.21.15.145:0/3754794222 conn(0x5596ada31090 msgr2=0x5596ada31490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.152+0000 7ffabe591740 1 -- 172.21.15.145:0/3754794222 shutdown_connections 2024-03-28T18:40:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.153+0000 7ffabe591740 1 -- 172.21.15.145:0/3754794222 wait complete. 2024-03-28T18:40:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.153+0000 7ffabe591740 1 -- start start 2024-03-28T18:40:13.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 Processor -- start 2024-03-28T18:40:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada3fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada3b680 0x5596ada47290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ada3e680 con 0x5596ada35b20 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ad956d50 con 0x5596ada3b680 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ada3e800 con 0x5596ada36510 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.154+0000 7ffabe591740 10 client.0 _ll_register_callbacks cb 0x5596ad956720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.156+0000 7ffab9f46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada3b680 0x5596ada47290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.156+0000 7ffab8f44640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.156+0000 7ffab8f44640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:59358/0 (socket says 172.21.15.145:59358) 2024-03-28T18:40:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.156+0000 7ffab8f44640 1 -- 172.21.15.145:0/3100917690 learned_addr learned my addr 172.21.15.145:0/3100917690 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:13.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.156+0000 7ffab9745640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada3fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:12 smithi192 ceph-mon[36589]: pgmap v424: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 722 MiB/s wr, 1.19k op/s 2024-03-28T18:40:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.157+0000 7ffab8f44640 1 -- 172.21.15.145:0/3100917690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada3b680 msgr2=0x5596ada47290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.157+0000 7ffab8f44640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada3b680 0x5596ada47290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.157+0000 7ffab8f44640 1 -- 172.21.15.145:0/3100917690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 msgr2=0x5596ada3fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.157+0000 7ffab8f44640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada3fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.157+0000 7ffab8f44640 1 -- 172.21.15.145:0/3100917690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596ad77b620 con 0x5596ada36510 2024-03-28T18:40:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.159+0000 7ffab8f44640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4005c20 tx=0x7ffaa40be170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f512c3987187611a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.159 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.159+0000 7ffa9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:40:13.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.160+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40d6070 con 0x5596ada36510 2024-03-28T18:40:13.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.160+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ffaa40efa80 con 0x5596ada36510 2024-03-28T18:40:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.165+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab0015920 con 0x5596ada36510 2024-03-28T18:40:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.166+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaa40505e0 con 0x5596ada36510 2024-03-28T18:40:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.166+0000 7ffabe591740 10 client.25534 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.166+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5596ada3e750 con 0x5596ada36510 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.167+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffab0015920 con 0x5596ada36510 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ffaa4050a10 con 0x5596ada36510 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 10 client.25534 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 20 client.25534 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596ada4bcd0 0x5596ada4e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 4 client.25534 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:40:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.168+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596ad9a8050 con 0x5596ada4bcd0 2024-03-28T18:40:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.169+0000 7ffab9745640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596ada4bcd0 0x5596ada4e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.174+0000 7ffab9745640 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596ada4bcd0 0x5596ada4e150 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a98f945d7c8335a server_cookie=c48044e9705ac20c in_seq=0 out_seq=0 2024-03-28T18:40:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.175+0000 7ffa9cff9640 10 client.25534 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.184+0000 7ffa9cff9640 1 -- 172.21.15.145:0/3100917690 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596ad9a8050 con 0x5596ada4bcd0 2024-03-28T18:40:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.184+0000 7ffa9cff9640 10 client.25534 handle_command_reply: tid=0 2024-03-28T18:40:13.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.184+0000 7ffabe591740 1 client.25534 shutdown 2024-03-28T18:40:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.189+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 msgr2=0x5596ada40250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.189+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4005c20 tx=0x7ffaa40be170 comp rx=0 tx=0).stop 2024-03-28T18:40:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 shutdown_connections 2024-03-28T18:40:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596ada4bcd0 0x5596ada4e150 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada3b680 0x5596ada47290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada40250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 --2- 172.21.15.145:0/3100917690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada3fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 >> 172.21.15.145:0/3100917690 conn(0x5596ada31090 msgr2=0x5596ada30290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.198+0000 7ffa9cff9640 0 client.25534 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.200+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 shutdown_connections 2024-03-28T18:40:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.201+0000 7ffabe591740 1 -- 172.21.15.145:0/3100917690 wait complete. 2024-03-28T18:40:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.201+0000 7ffabe591740 20 client.25534 trim_cache size 0 max 16384 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.235+0000 7ffabe591740 1 Processor -- start 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.235+0000 7ffabe591740 1 -- start start 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.235+0000 7ffabe591740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.235+0000 7ffabe591740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada36510 0x5596ada47250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.235+0000 7ffabe591740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada37b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffabe591740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800fae0 con 0x5596ada35b20 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffabe591740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98011460 con 0x5596ada36510 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffabe591740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98010140 con 0x5596ada3b680 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53642/0 (socket says 172.21.15.145:53642) 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 -- 172.21.15.145:0/2117230333 learned_addr learned my addr 172.21.15.145:0/2117230333 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9f46640 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada37b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab8f44640 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada36510 0x5596ada47250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 -- 172.21.15.145:0/2117230333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 msgr2=0x5596ada37b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada37b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 -- 172.21.15.145:0/2117230333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada36510 msgr2=0x5596ada47250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada36510 0x5596ada47250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.236+0000 7ffab9745640 1 -- 172.21.15.145:0/2117230333 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ada4e7d0 con 0x5596ada35b20 2024-03-28T18:40:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.237+0000 7ffab9745640 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0043b0 tx=0x7ffaac003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8497206929476ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.237+0000 7ffa9effd640 1 -- 172.21.15.145:0/2117230333 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004ab0 con 0x5596ada35b20 2024-03-28T18:40:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.237+0000 7ffa9effd640 1 -- 172.21.15.145:0/2117230333 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaac004c50 con 0x5596ada35b20 2024-03-28T18:40:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.237+0000 7ffa9effd640 1 -- 172.21.15.145:0/2117230333 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0127a0 con 0x5596ada35b20 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 -- 172.21.15.145:0/2117230333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 msgr2=0x5596ada375f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0043b0 tx=0x7ffaac003da0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 -- 172.21.15.145:0/2117230333 shutdown_connections 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada3b680 0x5596ada37b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada36510 0x5596ada47250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 --2- 172.21.15.145:0/2117230333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada35b20 0x5596ada375f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.239+0000 7ffabe591740 1 -- 172.21.15.145:0/2117230333 >> 172.21.15.145:0/2117230333 conn(0x5596ada31090 msgr2=0x5596ada3fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.241+0000 7ffabe591740 1 -- 172.21.15.145:0/2117230333 shutdown_connections 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.241+0000 7ffabe591740 1 -- 172.21.15.145:0/2117230333 wait complete. 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.242+0000 7ffabe591740 1 -- start start 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.242+0000 7ffabe591740 1 Processor -- start 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada45930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98010e00 con 0x5596ada3b680 2024-03-28T18:40:13.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980107a0 con 0x5596ada35b20 2024-03-28T18:40:13.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9800dca0 con 0x5596ada36510 2024-03-28T18:40:13.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.243+0000 7ffabe591740 10 client.0 _ll_register_callbacks cb 0x7ffaa40be5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.245+0000 7ffab9f46640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.245+0000 7ffab9f46640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53654/0 (socket says 172.21.15.145:53654) 2024-03-28T18:40:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.245+0000 7ffab9f46640 1 -- 172.21.15.145:0/1322643663 learned_addr learned my addr 172.21.15.145:0/1322643663 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:13.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.247+0000 7ffab8f44640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada45930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.247+0000 7ffab9745640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.248+0000 7ffab9f46640 1 -- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 msgr2=0x5596ada45930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.248+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada45930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.248+0000 7ffab9f46640 1 -- 172.21.15.145:0/1322643663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 msgr2=0x5596ada453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.248+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.248+0000 7ffab9f46640 1 -- 172.21.15.145:0/1322643663 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596ada46e40 con 0x5596ada3b680 2024-03-28T18:40:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.250+0000 7ffab9f46640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffab000a8f0 tx=0x7ffab0009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff98cdbf139bb3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.250+0000 7ffa9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:40:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.251+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0030070 con 0x5596ada3b680 2024-03-28T18:40:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.252+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ffab001f870 con 0x5596ada3b680 2024-03-28T18:40:13.252 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.252+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaac004c50 con 0x5596ada3b680 2024-03-28T18:40:13.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.252+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab003a050 con 0x5596ada3b680 2024-03-28T18:40:13.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.253+0000 7ffabe591740 10 client.15693 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:13.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.253+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5596adaa8060 con 0x5596ada3b680 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaac004c50 con 0x5596ada3b680 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ffab00355e0 con 0x5596ada3b680 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 10 client.15693 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 10 client.15693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 20 client.15693 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:13.254 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596adaa8780 0x5596adaa8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:13.255 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 4 client.15693 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:40:13.255 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.254+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5596ad9a23a0 con 0x5596adaa8780 2024-03-28T18:40:13.255 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.255+0000 7ffab9745640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596adaa8780 0x5596adaa8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:13.256 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.256+0000 7ffa9cff9640 10 client.15693 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:13.256 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:13.256+0000 7ffab9745640 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596adaa8780 0x5596adaa8b90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fa27d656b23e400 server_cookie=ca0258941804b67b in_seq=0 out_seq=0 2024-03-28T18:40:14.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:14 smithi164 ceph-mon[36248]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.14k op/s 2024-03-28T18:40:15.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:14 smithi145 ceph-mon[32151]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.14k op/s 2024-03-28T18:40:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:14 smithi192 ceph-mon[36589]: pgmap v425: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.14k op/s 2024-03-28T18:40:17.028 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:16 smithi145 ceph-mon[32151]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 912 MiB/s wr, 1.49k op/s 2024-03-28T18:40:17.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:16 smithi192 ceph-mon[36589]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 912 MiB/s wr, 1.49k op/s 2024-03-28T18:40:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:16 smithi164 ceph-mon[36248]: pgmap v426: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 912 MiB/s wr, 1.49k op/s 2024-03-28T18:40:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffa9cff9640 1 -- 172.21.15.145:0/1322643663 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5596ad9a23a0 con 0x5596adaa8780 2024-03-28T18:40:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffa9cff9640 10 client.15693 handle_command_reply: tid=0 2024-03-28T18:40:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 client.15693 shutdown 2024-03-28T18:40:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 msgr2=0x5596ada46780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffab000a8f0 tx=0x7ffab0009000 comp rx=0 tx=0).stop 2024-03-28T18:40:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 shutdown_connections 2024-03-28T18:40:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5596adaa8780 0x5596adaa8b90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ada3b680 0x5596ada46780 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5596ada36510 0x5596ada45930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 --2- 172.21.15.145:0/1322643663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5596ada35b20 0x5596ada453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffa9cff9640 0 client.15693 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.546+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 >> 172.21.15.145:0/1322643663 conn(0x5596ada31090 msgr2=0x5596ada3fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.547+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 shutdown_connections 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.547+0000 7ffabe591740 1 -- 172.21.15.145:0/1322643663 wait complete. 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.547+0000 7ffabe591740 20 client.15693 trim_cache size 0 max 16384 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:40:17.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:40:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.562+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 msgr2=0x7ffaa00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.562+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 0x7ffaa00a4ec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffab001c000 tx=0x7ffab001d740 comp rx=0 tx=0).stop 2024-03-28T18:40:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.563+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 msgr2=0x7ffab4197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.563+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009850 tx=0x7ffaa402c6b0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.563+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 shutdown_connections 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.563+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffab4107210 0x7ffab419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.564+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffab41048e0 0x7ffab4197e30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.564+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ffaa00a2a00 0x7ffaa00a4ec0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.564+0000 7ffaba747640 1 --2- 172.21.15.145:0/1526497246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab4101fb0 0x7ffab41978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.564+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 >> 172.21.15.145:0/1526497246 conn(0x7ffab40fb9a0 msgr2=0x7ffab4102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.565+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 shutdown_connections 2024-03-28T18:40:17.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:17.565+0000 7ffaba747640 1 -- 172.21.15.145:0/1526497246 wait complete. 2024-03-28T18:40:19.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:18 smithi145 ceph-mon[32151]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 693 MiB/s wr, 1.14k op/s 2024-03-28T18:40:19.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:18 smithi192 ceph-mon[36589]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 693 MiB/s wr, 1.14k op/s 2024-03-28T18:40:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:18 smithi164 ceph-mon[36248]: pgmap v427: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 693 MiB/s wr, 1.14k op/s 2024-03-28T18:40:21.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:20 smithi145 ceph-mon[32151]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 1.49k op/s 2024-03-28T18:40:21.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:20 smithi192 ceph-mon[36589]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 1.49k op/s 2024-03-28T18:40:21.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:20 smithi164 ceph-mon[36248]: pgmap v428: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 1.49k op/s 2024-03-28T18:40:23.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:22 smithi145 ceph-mon[32151]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 747 MiB/s wr, 1.42k op/s 2024-03-28T18:40:23.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:22 smithi192 ceph-mon[36589]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 747 MiB/s wr, 1.42k op/s 2024-03-28T18:40:23.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:22 smithi164 ceph-mon[36248]: pgmap v429: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 747 MiB/s wr, 1.42k op/s 2024-03-28T18:40:25.031 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:24 smithi145 ceph-mon[32151]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 681 MiB/s wr, 1.35k op/s 2024-03-28T18:40:25.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:24 smithi192 ceph-mon[36589]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 681 MiB/s wr, 1.35k op/s 2024-03-28T18:40:25.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:24 smithi164 ceph-mon[36248]: pgmap v430: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 681 MiB/s wr, 1.35k op/s 2024-03-28T18:40:27.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:26 smithi145 ceph-mon[32151]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 894 MiB/s wr, 1.80k op/s 2024-03-28T18:40:27.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:26 smithi192 ceph-mon[36589]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 894 MiB/s wr, 1.80k op/s 2024-03-28T18:40:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:26 smithi164 ceph-mon[36248]: pgmap v431: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 894 MiB/s wr, 1.80k op/s 2024-03-28T18:40:29.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:28 smithi192 ceph-mon[36589]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 695 MiB/s wr, 1.45k op/s 2024-03-28T18:40:29.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:28 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:29.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:28 smithi164 ceph-mon[36248]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 695 MiB/s wr, 1.45k op/s 2024-03-28T18:40:29.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:28 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:29.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:28 smithi145 ceph-mon[32151]: pgmap v432: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 695 MiB/s wr, 1.45k op/s 2024-03-28T18:40:29.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:28 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.773+0000 7f00c0797640 1 -- 172.21.15.145:0/1029193120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b81093d0 msgr2=0x7f00b810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.773+0000 7f00c0797640 1 --2- 172.21.15.145:0/1029193120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b81093d0 0x7f00b810e060 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009900 tx=0x7f00a802bc80 comp rx=0 tx=0).stop 2024-03-28T18:40:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.774+0000 7f00c0797640 1 -- 172.21.15.145:0/1029193120 shutdown_connections 2024-03-28T18:40:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.774+0000 7f00c0797640 1 --2- 172.21.15.145:0/1029193120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b81093d0 0x7f00b810e060 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.774+0000 7f00c0797640 1 --2- 172.21.15.145:0/1029193120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.774+0000 7f00c0797640 1 --2- 172.21.15.145:0/1029193120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b8072970 0x7f00b8072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.774+0000 7f00c0797640 1 -- 172.21.15.145:0/1029193120 >> 172.21.15.145:0/1029193120 conn(0x7f00b806e510 msgr2=0x7f00b8070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.778+0000 7f00c0797640 1 -- 172.21.15.145:0/1029193120 shutdown_connections 2024-03-28T18:40:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.778+0000 7f00c0797640 1 -- 172.21.15.145:0/1029193120 wait complete. 2024-03-28T18:40:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.779+0000 7f00c0797640 1 Processor -- start 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.779+0000 7f00c0797640 1 -- start start 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b8072970 0x7f00b8116cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b81093d0 0x7f00b81b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b811c0f0 con 0x7f00b81093d0 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b811bf70 con 0x7f00b8072970 2024-03-28T18:40:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00c0797640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b811c270 con 0x7f00b80732d0 2024-03-28T18:40:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00bdd32640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00bdd32640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:59020/0 (socket says 172.21.15.145:59020) 2024-03-28T18:40:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.780+0000 7f00bdd32640 1 -- 172.21.15.145:0/3708388489 learned_addr learned my addr 172.21.15.145:0/3708388489 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00be533640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b8072970 0x7f00b8116cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bed34640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b81093d0 0x7f00b81b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bdd32640 1 -- 172.21.15.145:0/3708388489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b8072970 msgr2=0x7f00b8116cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bdd32640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b8072970 0x7f00b8116cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bdd32640 1 -- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b81093d0 msgr2=0x7f00b81b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bdd32640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b81093d0 0x7f00b81b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.781+0000 7f00bdd32640 1 -- 172.21.15.145:0/3708388489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b81178a0 con 0x7f00b80732d0 2024-03-28T18:40:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.782+0000 7f00bdd32640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f00b000b9c0 tx=0x7f00b000bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.786+0000 7f00af7fe640 1 -- 172.21.15.145:0/3708388489 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b0019070 con 0x7f00b80732d0 2024-03-28T18:40:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.786+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b81b6e00 con 0x7f00b80732d0 2024-03-28T18:40:30.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.786+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b81b72e0 con 0x7f00b80732d0 2024-03-28T18:40:30.788 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.787+0000 7f00af7fe640 1 -- 172.21.15.145:0/3708388489 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00b0014440 con 0x7f00b80732d0 2024-03-28T18:40:30.788 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.787+0000 7f00af7fe640 1 -- 172.21.15.145:0/3708388489 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b00125a0 con 0x7f00b80732d0 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.790+0000 7f00c3385740 1 Processor -- start 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.792+0000 7f00c3385740 1 -- start start 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.793+0000 7f00c3385740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00c3385740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x564708184010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00c3385740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x564708186b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00c3385740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647080e3b10 con 0x564708184680 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00c3385740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56470808c900 con 0x56470817f510 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00c3385740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647080dfbb0 con 0x56470817eb20 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00af7fe640 1 -- 172.21.15.145:0/3708388489 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f00b001c070 con 0x7f00b80732d0 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00bed34640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00bed34640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:59030/0 (socket says 172.21.15.145:59030) 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.795+0000 7f00bed34640 1 -- 172.21.15.145:0/462081781 learned_addr learned my addr 172.21.15.145:0/462081781 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:30.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.796+0000 7f00be533640 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x564708184010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 -- 172.21.15.145:0/462081781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 msgr2=0x564708184010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x564708184010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 -- 172.21.15.145:0/462081781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 msgr2=0x564708186b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x564708186b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 -- 172.21.15.145:0/462081781 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564707df8210 con 0x56470817eb20 2024-03-28T18:40:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00af7fe640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 0x7f00a40a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.797+0000 7f00bed34640 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f00a8006010 tx=0x7f00a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e36d09b3097f2229 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.798+0000 7f00af7fe640 1 -- 172.21.15.145:0/3708388489 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f00b00ce1f0 con 0x7f00b80732d0 2024-03-28T18:40:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.799+0000 7f00acff9640 1 -- 172.21.15.145:0/462081781 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a805a040 con 0x56470817eb20 2024-03-28T18:40:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.799+0000 7f00bed34640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 0x7f00a40a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.799+0000 7f00acff9640 1 -- 172.21.15.145:0/462081781 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8064020 con 0x56470817eb20 2024-03-28T18:40:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.799+0000 7f00acff9640 1 -- 172.21.15.145:0/462081781 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8063040 con 0x56470817eb20 2024-03-28T18:40:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.800+0000 7f00c3385740 1 -- 172.21.15.145:0/462081781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 msgr2=0x56470817ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.800+0000 7f00bed34640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 0x7f00a40a4e80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00a8061000 tx=0x7f00a80039c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.800+0000 7f00c3385740 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f00a8006010 tx=0x7f00a80046c0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.801+0000 7f00c3385740 1 -- 172.21.15.145:0/462081781 shutdown_connections 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.801+0000 7f00c3385740 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x564708186b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.801+0000 7f00c3385740 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x564708184010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.801+0000 7f00c3385740 1 --2- 172.21.15.145:0/462081781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x56470817ef40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.801+0000 7f00c3385740 1 -- 172.21.15.145:0/462081781 >> 172.21.15.145:0/462081781 conn(0x56470817a090 msgr2=0x56470817a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.802+0000 7f00c3385740 1 -- 172.21.15.145:0/462081781 shutdown_connections 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.802+0000 7f00c3385740 1 -- 172.21.15.145:0/462081781 wait complete. 2024-03-28T18:40:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 -- start start 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 Processor -- start 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x5647081893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x5647081903c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647081875d0 con 0x564708184680 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56470809fd50 con 0x56470817f510 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.803+0000 7f00c3385740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564708187750 con 0x56470817eb20 2024-03-28T18:40:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00c3385740 10 client.0 _ll_register_callbacks cb 0x56470809f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00bed34640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x5647081903c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00be533640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00be533640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:59036/0 (socket says 172.21.15.145:59036) 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00be533640 1 -- 172.21.15.145:0/1838955780 learned_addr learned my addr 172.21.15.145:0/1838955780 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.804+0000 7f00bdd32640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x5647081893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.805+0000 7f00be533640 1 -- 172.21.15.145:0/1838955780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 msgr2=0x5647081893f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.805+0000 7f00be533640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x5647081893f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.805+0000 7f00be533640 1 -- 172.21.15.145:0/1838955780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 msgr2=0x5647081903c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.805+0000 7f00be533640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x5647081903c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.805+0000 7f00be533640 1 -- 172.21.15.145:0/1838955780 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564707ec4620 con 0x56470817eb20 2024-03-28T18:40:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.806+0000 7f00be533640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009f00 tx=0x7f00b4011630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4b42e014fa2856c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.821+0000 7f00a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:40:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.821+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4019070 con 0x56470817eb20 2024-03-28T18:40:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.821+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f00b4015660 con 0x56470817eb20 2024-03-28T18:40:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.821+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00a8064020 con 0x56470817eb20 2024-03-28T18:40:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.839+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00b4023050 con 0x56470817eb20 2024-03-28T18:40:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.839+0000 7f00c3385740 10 client.25549 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.839+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564708193800 con 0x56470817eb20 2024-03-28T18:40:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.841+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8064020 con 0x56470817eb20 2024-03-28T18:40:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.843+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f00b401eb90 con 0x56470817eb20 2024-03-28T18:40:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 10 client.25549 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 20 client.25549 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708194ed0 0x5647081972e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 4 client.25549 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:40:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.844+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647080f1050 con 0x564708194ed0 2024-03-28T18:40:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.845+0000 7f00bdd32640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708194ed0 0x5647081972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.849+0000 7f00bdd32640 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708194ed0 0x5647081972e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8ffa7042c7c016 server_cookie=dd86884260a0de71 in_seq=0 out_seq=0 2024-03-28T18:40:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.849+0000 7f00a1ffb640 10 client.25549 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.857+0000 7f00a1ffb640 1 -- 172.21.15.145:0/1838955780 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5647080f1050 con 0x564708194ed0 2024-03-28T18:40:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.857+0000 7f00a1ffb640 10 client.25549 handle_command_reply: tid=0 2024-03-28T18:40:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.858+0000 7f00c3385740 1 client.25549 shutdown 2024-03-28T18:40:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 msgr2=0x564708188eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009f00 tx=0x7f00b4011630 comp rx=0 tx=0).stop 2024-03-28T18:40:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 shutdown_connections 2024-03-28T18:40:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708194ed0 0x5647081972e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564708184680 0x5647081903c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817f510 0x5647081893f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 --2- 172.21.15.145:0/1838955780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817eb20 0x564708188eb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00a1ffb640 0 client.25549 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 >> 172.21.15.145:0/1838955780 conn(0x56470817a090 msgr2=0x564708179290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.859+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 shutdown_connections 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.860+0000 7f00c3385740 1 -- 172.21.15.145:0/1838955780 wait complete. 2024-03-28T18:40:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.860+0000 7f00c3385740 20 client.25549 trim_cache size 0 max 16384 2024-03-28T18:40:30.904 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.903+0000 7f00c3385740 1 Processor -- start 2024-03-28T18:40:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.904+0000 7f00c3385740 1 -- start start 2024-03-28T18:40:30.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.904+0000 7f00c3385740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00be533640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00be533640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43184/0 (socket says 172.21.15.145:43184) 2024-03-28T18:40:30.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00be533640 1 -- 172.21.15.145:0/152136619 learned_addr learned my addr 172.21.15.145:0/152136619 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:30.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x564708178f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817f510 0x564708179480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.905+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00f9a0 con 0x56470817a090 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.906+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c011320 con 0x56470817eb20 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.906+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c010000 con 0x56470817f510 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.906+0000 7f00bdd32640 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x564708178f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.906+0000 7f00bed34640 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817f510 0x564708179480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.907+0000 7f00be533640 1 -- 172.21.15.145:0/152136619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817f510 msgr2=0x564708179480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.907+0000 7f00be533640 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817f510 0x564708179480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.908+0000 7f00be533640 1 -- 172.21.15.145:0/152136619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 msgr2=0x564708178f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.908+0000 7f00be533640 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x564708178f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.908+0000 7f00be533640 1 -- 172.21.15.145:0/152136619 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56470817ef00 con 0x56470817a090 2024-03-28T18:40:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.908+0000 7f00be533640 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b4011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d495378bce921d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.909+0000 7f00a3fff640 1 -- 172.21.15.145:0/152136619 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4019040 con 0x56470817a090 2024-03-28T18:40:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.909+0000 7f00a3fff640 1 -- 172.21.15.145:0/152136619 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00b401eb20 con 0x56470817a090 2024-03-28T18:40:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.910+0000 7f00a3fff640 1 -- 172.21.15.145:0/152136619 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4015460 con 0x56470817a090 2024-03-28T18:40:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.910+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 msgr2=0x564708178a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.910+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000c00 tx=0x7f00b4011be0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.910+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 shutdown_connections 2024-03-28T18:40:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817f510 0x564708179480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x564708178f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 --2- 172.21.15.145:0/152136619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817a090 0x564708178a00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 >> 172.21.15.145:0/152136619 conn(0x564708180110 msgr2=0x56470817a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 shutdown_connections 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 -- 172.21.15.145:0/152136619 wait complete. 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.911+0000 7f00c3385740 1 -- start start 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 Processor -- start 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817a090 0x56470818d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x56470818dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c010cc0 con 0x56470817f510 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c010660 con 0x56470817eb20 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c00dc10 con 0x56470817a090 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.912+0000 7f00c3385740 10 client.0 _ll_register_callbacks cb 0x7f00b4009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.913+0000 7f00bed34640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.913+0000 7f00bed34640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43194/0 (socket says 172.21.15.145:43194) 2024-03-28T18:40:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.913+0000 7f00bed34640 1 -- 172.21.15.145:0/2612865080 learned_addr learned my addr 172.21.15.145:0/2612865080 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.913+0000 7f00bdd32640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x56470818dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.914+0000 7f00bed34640 1 -- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817a090 msgr2=0x56470818d6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.914+0000 7f00bed34640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817a090 0x56470818d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.914+0000 7f00bed34640 1 -- 172.21.15.145:0/2612865080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 msgr2=0x56470818dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.914+0000 7f00bed34640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x56470818dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.914+0000 7f00bed34640 1 -- 172.21.15.145:0/2612865080 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56470818f210 con 0x56470817f510 2024-03-28T18:40:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.916+0000 7f00bed34640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c290 tx=0x7f00a8066000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa6e19b0bb2fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.917+0000 7f00a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:40:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.920+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a80052e0 con 0x56470817f510 2024-03-28T18:40:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.920+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00b401eb20 con 0x56470817f510 2024-03-28T18:40:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.920+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f00a802c620 con 0x56470817f510 2024-03-28T18:40:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.921+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a8087050 con 0x56470817f510 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.921+0000 7f00c3385740 10 client.15702 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.921+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5647081f0c60 con 0x56470817f510 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00b401eb20 con 0x56470817f510 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f00a806c660 con 0x56470817f510 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 10 client.15702 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 20 client.15702 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708184680 0x5647081f3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 4 client.15702 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:40:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.922+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5647080eb3a0 con 0x564708184680 2024-03-28T18:40:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.929+0000 7f00be533640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708184680 0x5647081f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.934+0000 7f00a1ffb640 10 client.15702 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:30.934+0000 7f00be533640 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708184680 0x5647081f3620 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d197bfa13d07c6b server_cookie=d18d64cfed7a858 in_seq=0 out_seq=0 2024-03-28T18:40:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:30 smithi192 ceph-mon[36589]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 MiB/s wr, 1.91k op/s 2024-03-28T18:40:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:30 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:40:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:30 smithi164 ceph-mon[36248]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 MiB/s wr, 1.91k op/s 2024-03-28T18:40:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:30 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:40:31.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:30 smithi145 ceph-mon[32151]: pgmap v433: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 846 MiB/s wr, 1.91k op/s 2024-03-28T18:40:31.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:31.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:31.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:30 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:40:33.078 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:32 smithi145 ceph-mon[32151]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 714 MiB/s wr, 1.74k op/s 2024-03-28T18:40:33.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:32 smithi192 ceph-mon[36589]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 714 MiB/s wr, 1.74k op/s 2024-03-28T18:40:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:32 smithi164 ceph-mon[36248]: pgmap v434: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 714 MiB/s wr, 1.74k op/s 2024-03-28T18:40:35.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:34 smithi192 ceph-mon[36589]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 655 MiB/s wr, 1.56k op/s 2024-03-28T18:40:35.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:34 smithi164 ceph-mon[36248]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 655 MiB/s wr, 1.56k op/s 2024-03-28T18:40:35.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:34 smithi145 ceph-mon[32151]: pgmap v435: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 655 MiB/s wr, 1.56k op/s 2024-03-28T18:40:36.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:35 smithi192 ceph-mon[36589]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 849 MiB/s wr, 1.96k op/s 2024-03-28T18:40:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:35 smithi164 ceph-mon[36248]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 849 MiB/s wr, 1.96k op/s 2024-03-28T18:40:36.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:35 smithi145 ceph-mon[32151]: pgmap v436: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 849 MiB/s wr, 1.96k op/s 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.580+0000 7f00a1ffb640 1 -- 172.21.15.145:0/2612865080 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5647080eb3a0 con 0x564708184680 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.580+0000 7f00a1ffb640 10 client.15702 handle_command_reply: tid=0 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.580+0000 7f00c3385740 1 client.15702 shutdown 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 msgr2=0x56470818ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c290 tx=0x7f00a8066000 comp rx=0 tx=0).stop 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 shutdown_connections 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x564708184680 0x5647081f3620 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56470817f510 0x56470818ec20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56470817eb20 0x56470818dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 --2- 172.21.15.145:0/2612865080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56470817a090 0x56470818d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 >> 172.21.15.145:0/2612865080 conn(0x564708180110 msgr2=0x56470817d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.582+0000 7f00a1ffb640 0 client.15702 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.583+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 shutdown_connections 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.583+0000 7f00c3385740 1 -- 172.21.15.145:0/2612865080 wait complete. 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:40:37.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:40:37.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.583+0000 7f00c3385740 20 client.15702 trim_cache size 0 max 16384 2024-03-28T18:40:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.589+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 msgr2=0x7f00a40a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.589+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 0x7f00a40a4e80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00a8061000 tx=0x7f00a80039c0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.589+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 msgr2=0x7f00b8117230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.589+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f00b000b9c0 tx=0x7f00b000bdc0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 shutdown_connections 2024-03-28T18:40:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b81093d0 0x7f00b81b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00a40a29c0 0x7f00a40a4e80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00b80732d0 0x7f00b8117230 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 --2- 172.21.15.145:0/3708388489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00b8072970 0x7f00b8116cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.597+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 >> 172.21.15.145:0/3708388489 conn(0x7f00b806e510 msgr2=0x7f00b810c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.598+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 shutdown_connections 2024-03-28T18:40:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:37.598+0000 7f00c0797640 1 -- 172.21.15.145:0/3708388489 wait complete. 2024-03-28T18:40:38.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:38 smithi192 ceph-mon[36589]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 642 MiB/s wr, 1.57k op/s 2024-03-28T18:40:38.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:38 smithi164 ceph-mon[36248]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 642 MiB/s wr, 1.57k op/s 2024-03-28T18:40:39.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:38 smithi145 ceph-mon[32151]: pgmap v437: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 642 MiB/s wr, 1.57k op/s 2024-03-28T18:40:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:40 smithi164 ceph-mon[36248]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.74k op/s 2024-03-28T18:40:41.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:40 smithi145 ceph-mon[32151]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.74k op/s 2024-03-28T18:40:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:40 smithi192 ceph-mon[36589]: pgmap v438: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.74k op/s 2024-03-28T18:40:42.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:42 smithi164 ceph-mon[36248]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 690 MiB/s wr, 1.40k op/s 2024-03-28T18:40:43.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:42 smithi145 ceph-mon[32151]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 690 MiB/s wr, 1.40k op/s 2024-03-28T18:40:43.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:42 smithi192 ceph-mon[36589]: pgmap v439: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 690 MiB/s wr, 1.40k op/s 2024-03-28T18:40:44.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:44 smithi164 ceph-mon[36248]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 645 MiB/s wr, 1.21k op/s 2024-03-28T18:40:45.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:44 smithi145 ceph-mon[32151]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 645 MiB/s wr, 1.21k op/s 2024-03-28T18:40:45.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:44 smithi192 ceph-mon[36589]: pgmap v440: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 645 MiB/s wr, 1.21k op/s 2024-03-28T18:40:46.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:46 smithi164 ceph-mon[36248]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 834 MiB/s wr, 1.59k op/s 2024-03-28T18:40:47.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:46 smithi145 ceph-mon[32151]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 834 MiB/s wr, 1.59k op/s 2024-03-28T18:40:47.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:46 smithi192 ceph-mon[36589]: pgmap v441: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 834 MiB/s wr, 1.59k op/s 2024-03-28T18:40:48.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:48 smithi164 ceph-mon[36248]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-03-28T18:40:49.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:48 smithi145 ceph-mon[32151]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-03-28T18:40:49.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:48 smithi192 ceph-mon[36589]: pgmap v442: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 708 MiB/s wr, 1.30k op/s 2024-03-28T18:40:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.851+0000 7f63aab37640 1 -- 172.21.15.145:0/2581325196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a410d9c0 msgr2=0x7f63a410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.851+0000 7f63aab37640 1 --2- 172.21.15.145:0/2581325196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a410d9c0 0x7f63a410de40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f639c009ca0 tx=0x7f639c02c120 comp rx=0 tx=0).stop 2024-03-28T18:40:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.853+0000 7f63aab37640 1 -- 172.21.15.145:0/2581325196 shutdown_connections 2024-03-28T18:40:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.853+0000 7f63aab37640 1 --2- 172.21.15.145:0/2581325196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a410d9c0 0x7f63a410de40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.853+0000 7f63aab37640 1 --2- 172.21.15.145:0/2581325196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410b010 0x7f63a410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.853+0000 7f63aab37640 1 --2- 172.21.15.145:0/2581325196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a41086a0 0x7f63a410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.853+0000 7f63aab37640 1 -- 172.21.15.145:0/2581325196 >> 172.21.15.145:0/2581325196 conn(0x7f63a406e6e0 msgr2=0x7f63a4070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.857+0000 7f63aab37640 1 -- 172.21.15.145:0/2581325196 shutdown_connections 2024-03-28T18:40:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.857+0000 7f63aab37640 1 -- 172.21.15.145:0/2581325196 wait complete. 2024-03-28T18:40:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.858+0000 7f63aab37640 1 Processor -- start 2024-03-28T18:40:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 -- start start 2024-03-28T18:40:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a41086a0 0x7f63a4114bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a410b010 0x7f63a4115130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a411c1f0 con 0x7f63a410d9c0 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a411c070 con 0x7f63a41086a0 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.859+0000 7f63aab37640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a411c370 con 0x7f63a410b010 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.860+0000 7f63a90d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.860+0000 7f63a90d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35882/0 (socket says 172.21.15.145:35882) 2024-03-28T18:40:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.860+0000 7f63a90d4640 1 -- 172.21.15.145:0/3941196821 learned_addr learned my addr 172.21.15.145:0/3941196821 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f639bfff640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a410b010 0x7f63a4115130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f63a90d4640 1 -- 172.21.15.145:0/3941196821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a410b010 msgr2=0x7f63a4115130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f63a90d4640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a410b010 0x7f63a4115130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f63a90d4640 1 -- 172.21.15.145:0/3941196821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a41086a0 msgr2=0x7f63a4114bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f63a90d4640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a41086a0 0x7f63a4114bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.861+0000 7f63a90d4640 1 -- 172.21.15.145:0/3941196821 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4118260 con 0x7f63a410d9c0 2024-03-28T18:40:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.862+0000 7f63a90d4640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f639c037040 tx=0x7f639c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.862+0000 7f6399ffb640 1 -- 172.21.15.145:0/3941196821 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c039040 con 0x7f63a410d9c0 2024-03-28T18:40:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.863+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a41b6e40 con 0x7f63a410d9c0 2024-03-28T18:40:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.863+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a41b71f0 con 0x7f63a410d9c0 2024-03-28T18:40:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.868+0000 7f63ad7f8740 1 Processor -- start 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.868+0000 7f63ad7f8740 1 -- start start 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3b9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d319b10 con 0x56508d3b5310 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d2c2900 con 0x56508d3b4920 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.869+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d315bb0 con 0x56508d3ba480 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.870+0000 7f6399ffb640 1 -- 172.21.15.145:0/3941196821 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f639c044070 con 0x7f63a410d9c0 2024-03-28T18:40:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.870+0000 7f6399ffb640 1 -- 172.21.15.145:0/3941196821 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c03f440 con 0x7f63a410d9c0 2024-03-28T18:40:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.872+0000 7f639bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.872+0000 7f639bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.872+0000 7f639bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:48584/0 (socket says 172.21.15.145:48584) 2024-03-28T18:40:50.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.872+0000 7f639bfff640 1 -- 172.21.15.145:0/3753399312 learned_addr learned my addr 172.21.15.145:0/3753399312 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:50.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.872+0000 7f63a88d3640 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3b9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.875+0000 7f6399ffb640 1 -- 172.21.15.145:0/3941196821 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f639c03f5e0 con 0x7f63a410d9c0 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.876+0000 7f6399ffb640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 0x7f638c0a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f639bfff640 1 -- 172.21.15.145:0/3753399312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 msgr2=0x56508d3b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f639bfff640 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f639bfff640 1 -- 172.21.15.145:0/3753399312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 msgr2=0x56508d3b9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f639bfff640 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3b9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f639bfff640 1 -- 172.21.15.145:0/3753399312 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508d02e210 con 0x56508d3ba480 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f63a88d3640 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3b9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.877+0000 7f6399ffb640 1 -- 172.21.15.145:0/3941196821 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f639c0f1f70 con 0x7f63a410d9c0 2024-03-28T18:40:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.878+0000 7f639bfff640 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009460 tx=0x7f63a0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1591f87dee8786b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.878+0000 7f638affd640 1 -- 172.21.15.145:0/3753399312 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0005020 con 0x56508d3ba480 2024-03-28T18:40:50.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.879+0000 7f63a90d4640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 0x7f638c0a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.887 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.885+0000 7f638affd640 1 -- 172.21.15.145:0/3753399312 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63a00047a0 con 0x56508d3ba480 2024-03-28T18:40:50.887 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.885+0000 7f638affd640 1 -- 172.21.15.145:0/3753399312 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a00119f0 con 0x56508d3ba480 2024-03-28T18:40:50.890 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.891+0000 7f63ad7f8740 1 -- 172.21.15.145:0/3753399312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 msgr2=0x56508d3bc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.891+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009460 tx=0x7f63a0005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.894+0000 7f63ad7f8740 1 -- 172.21.15.145:0/3753399312 shutdown_connections 2024-03-28T18:40:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.894+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3bc910 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.894+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3b9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.894+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/3753399312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.894+0000 7f63ad7f8740 1 -- 172.21.15.145:0/3753399312 >> 172.21.15.145:0/3753399312 conn(0x56508d3afe90 msgr2=0x56508d3b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.899+0000 7f63a90d4640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 0x7f638c0a4f90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f639c033000 tx=0x7f639c02c120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.901+0000 7f63ad7f8740 1 -- 172.21.15.145:0/3753399312 shutdown_connections 2024-03-28T18:40:50.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.906+0000 7f63ad7f8740 1 -- 172.21.15.145:0/3753399312 wait complete. 2024-03-28T18:40:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.906+0000 7f63ad7f8740 1 -- start start 2024-03-28T18:40:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.906+0000 7f63ad7f8740 1 Processor -- start 2024-03-28T18:40:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3bf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d3bd430 con 0x56508d3b5310 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d2d5d50 con 0x56508d3b4920 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508d3bd5b0 con 0x56508d3ba480 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f63ad7f8740 10 client.0 _ll_register_callbacks cb 0x56508d2d5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.907+0000 7f639bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:59270/0 (socket says 172.21.15.145:59270) 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 -- 172.21.15.145:0/2059792019 learned_addr learned my addr 172.21.15.145:0/2059792019 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 -- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 msgr2=0x56508d3c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 -- 172.21.15.145:0/2059792019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 msgr2=0x56508d3bf1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:40:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3bf1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.908+0000 7f639bfff640 1 -- 172.21.15.145:0/2059792019 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56508d0fa620 con 0x56508d3b4920 2024-03-28T18:40:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.913+0000 7f639bfff640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f63a0005d80 tx=0x7f63a0009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=797ac780f1967324 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.913+0000 7f6383fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:40:50.916 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.917+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0005230 con 0x56508d3b4920 2024-03-28T18:40:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.917+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63a00047a0 con 0x56508d3b4920 2024-03-28T18:40:50.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.917+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f63a00053d0 con 0x56508d3b4920 2024-03-28T18:40:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.921+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a0036050 con 0x56508d3b4920 2024-03-28T18:40:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.921+0000 7f63ad7f8740 10 client.25463 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:50.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.921+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56508d3c94f0 con 0x56508d3b4920 2024-03-28T18:40:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.926+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63a00047a0 con 0x56508d3b4920 2024-03-28T18:40:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.926+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f63a00125f0 con 0x56508d3b4920 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 10 client.25463 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 10 client.25463 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 20 client.25463 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d3caf60 0x56508d3cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 4 client.25463 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:40:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.927+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56508d327050 con 0x56508d3caf60 2024-03-28T18:40:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.929+0000 7f63a88d3640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d3caf60 0x56508d3cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.941+0000 7f63a88d3640 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d3caf60 0x56508d3cd3a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53aaa386810274cc server_cookie=c99e01f6a2b1140d in_seq=0 out_seq=0 2024-03-28T18:40:50.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.941+0000 7f6383fff640 10 client.25463 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.945+0000 7f6383fff640 1 -- 172.21.15.145:0/2059792019 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56508d327050 con 0x56508d3caf60 2024-03-28T18:40:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.945+0000 7f6383fff640 10 client.25463 handle_command_reply: tid=0 2024-03-28T18:40:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.946+0000 7f63ad7f8740 1 client.25463 shutdown 2024-03-28T18:40:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.947+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 msgr2=0x56508d3bec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.947+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f63a0005d80 tx=0x7f63a0009370 comp rx=0 tx=0).stop 2024-03-28T18:40:50.950 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 shutdown_connections 2024-03-28T18:40:50.950 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d3caf60 0x56508d3cd3a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3bf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/2059792019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3b4920 0x56508d3bec90 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f63a0005d80 tx=0x7f63a0009370 comp rx=0 tx=0).stop 2024-03-28T18:40:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.948+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 >> 172.21.15.145:0/2059792019 conn(0x56508d3afe90 msgr2=0x56508d3af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.951+0000 7f6383fff640 0 client.25463 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:50.957 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.953+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 shutdown_connections 2024-03-28T18:40:50.957 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.953+0000 7f63ad7f8740 1 -- 172.21.15.145:0/2059792019 wait complete. 2024-03-28T18:40:50.957 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.953+0000 7f63ad7f8740 20 client.25463 trim_cache size 0 max 16384 2024-03-28T18:40:50.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:50 smithi164 ceph-mon[36248]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.39k op/s 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.995+0000 7f63ad7f8740 1 Processor -- start 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.995+0000 7f63ad7f8740 1 -- start start 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 0x56508d3b9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 0x56508d3c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3d2cb0 0x56508d3b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639000fb30 con 0x56508d3d2cb0 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900114b0 con 0x56508d3ba480 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f63ad7f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390010190 con 0x56508d3b5310 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f639bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 0x56508d3b9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f639bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 0x56508d3b9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:48604/0 (socket says 172.21.15.145:48604) 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f639bfff640 1 -- 172.21.15.145:0/4041598057 learned_addr learned my addr 172.21.15.145:0/4041598057 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.996+0000 7f639bfff640 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3d2cb0 0x56508d3b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 0x56508d3c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 -- 172.21.15.145:0/4041598057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 msgr2=0x56508d3b9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 0x56508d3b9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 -- 172.21.15.145:0/4041598057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3d2cb0 msgr2=0x56508d3b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3d2cb0 0x56508d3b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 -- 172.21.15.145:0/4041598057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508d3b98b0 con 0x56508d3ba480 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.997+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 0x56508d3c61a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x56508d3cdc20 tx=0x7f639400a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dff4e99890584771 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.998+0000 7f638a7fc640 1 -- 172.21.15.145:0/4041598057 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63940170f0 con 0x56508d3ba480 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.998+0000 7f638a7fc640 1 -- 172.21.15.145:0/4041598057 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f639401b030 con 0x56508d3ba480 2024-03-28T18:40:50.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:50.998+0000 7f638a7fc640 1 -- 172.21.15.145:0/4041598057 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639401f630 con 0x56508d3ba480 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.000+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4041598057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 msgr2=0x56508d3c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 0x56508d3c61a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x56508d3cdc20 tx=0x7f639400a570 comp rx=0 tx=0).stop 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4041598057 shutdown_connections 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3d2cb0 0x56508d3b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3ba480 0x56508d3c61a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4041598057 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3b5310 0x56508d3b9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.001 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.001+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4041598057 >> 172.21.15.145:0/4041598057 conn(0x56508d3afe90 msgr2=0x56508d3aaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.004+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4041598057 shutdown_connections 2024-03-28T18:40:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.004+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4041598057 wait complete. 2024-03-28T18:40:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.005+0000 7f63ad7f8740 1 -- start start 2024-03-28T18:40:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.005+0000 7f63ad7f8740 1 Processor -- start 2024-03-28T18:40:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f63ad7f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f639bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f639bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3c33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35904/0 (socket says 172.21.15.145:35904) 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f639bfff640 1 -- 172.21.15.145:0/4193219769 learned_addr learned my addr 172.21.15.145:0/4193219769 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 0x56508d424860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.006+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390010e50 con 0x56508d3b5310 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.007+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900107f0 con 0x56508d3d2cb0 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.007+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639000dcf0 con 0x56508d3ba480 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.007+0000 7f63ad7f8740 10 client.0 _ll_register_callbacks cb 0x7f63a0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:40:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.007+0000 7f639bfff640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 0x56508d424860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.009+0000 7f639bfff640 1 -- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 msgr2=0x56508d3c38f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:40:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.009+0000 7f639bfff640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c38f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.009+0000 7f639bfff640 1 -- 172.21.15.145:0/4193219769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 msgr2=0x56508d3c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:40:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.009+0000 7f639bfff640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.009+0000 7f639bfff640 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56508d424da0 con 0x56508d3d2cb0 2024-03-28T18:40:51.010 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.010+0000 7f639bfff640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 0x56508d424860 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f63a0014440 tx=0x7f63a0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa821861dc18f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:51.010 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.010+0000 7f6383fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:40:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.015+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0009850 con 0x56508d3d2cb0 2024-03-28T18:40:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.015+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f639401b030 con 0x56508d3d2cb0 2024-03-28T18:40:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.015+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f63a0012660 con 0x56508d3d2cb0 2024-03-28T18:40:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.016+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a0004020 con 0x56508d3d2cb0 2024-03-28T18:40:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.016+0000 7f63ad7f8740 10 client.25469 fetch_fsmap learned FSMap version 29 2024-03-28T18:40:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.016+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56508d425ed0 con 0x56508d3d2cb0 2024-03-28T18:40:51.017 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f639401b030 con 0x56508d3d2cb0 2024-03-28T18:40:51.017 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f63a0039040 con 0x56508d3d2cb0 2024-03-28T18:40:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 10 client.25469 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:40:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 10 client.25469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:40:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 20 client.25469 populate_metadata read hostname 'smithi145' 2024-03-28T18:40:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d426800 0x56508d428c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 4 client.25469 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:40:51.019 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.017+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56508d3213a0 con 0x56508d426800 2024-03-28T18:40:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.022+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d426800 0x56508d428c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.023+0000 7f63a88d3640 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d426800 0x56508d428c20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cbb3cb3fe3203ee server_cookie=6364f9d36708de30 in_seq=0 out_seq=0 2024-03-28T18:40:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:51.024+0000 7f6383fff640 10 client.25469 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:51.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:50 smithi145 ceph-mon[32151]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.39k op/s 2024-03-28T18:40:51.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:50 smithi192 ceph-mon[36589]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.39k op/s 2024-03-28T18:40:53.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:52 smithi145 ceph-mon[32151]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.29k op/s 2024-03-28T18:40:53.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:52 smithi192 ceph-mon[36589]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.29k op/s 2024-03-28T18:40:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:52 smithi164 ceph-mon[36248]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.29k op/s 2024-03-28T18:40:55.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:54 smithi145 ceph-mon[32151]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 MiB/s wr, 1.32k op/s 2024-03-28T18:40:55.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:54 smithi192 ceph-mon[36589]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 MiB/s wr, 1.32k op/s 2024-03-28T18:40:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:54 smithi164 ceph-mon[36248]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 MiB/s wr, 1.32k op/s 2024-03-28T18:40:57.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:56 smithi145 ceph-mon[32151]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.83k op/s 2024-03-28T18:40:57.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:56 smithi192 ceph-mon[36589]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.83k op/s 2024-03-28T18:40:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:56 smithi164 ceph-mon[36248]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.83k op/s 2024-03-28T18:40:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.490+0000 7f6383fff640 1 -- 172.21.15.145:0/4193219769 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56508d3213a0 con 0x56508d426800 2024-03-28T18:40:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.490+0000 7f6383fff640 10 client.25469 handle_command_reply: tid=0 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.490+0000 7f63ad7f8740 1 client.25469 shutdown 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 msgr2=0x56508d424860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 0x56508d424860 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f63a0014440 tx=0x7f63a0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 shutdown_connections 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56508d426800 0x56508d428c20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56508d3d2cb0 0x56508d424860 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56508d3ba480 0x56508d3c38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 --2- 172.21.15.145:0/4193219769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56508d3b5310 0x56508d3c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 >> 172.21.15.145:0/4193219769 conn(0x56508d3afe90 msgr2=0x56508d3b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f6383fff640 0 client.25469 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:40:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 shutdown_connections 2024-03-28T18:40:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.491+0000 7f63ad7f8740 1 -- 172.21.15.145:0/4193219769 wait complete. 2024-03-28T18:40:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.492+0000 7f63ad7f8740 20 client.25469 trim_cache size 0 max 16384 2024-03-28T18:40:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:40:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:40:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 msgr2=0x7f638c0a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 0x7f638c0a4f90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f639c033000 tx=0x7f639c02c120 comp rx=0 tx=0).stop 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 msgr2=0x7f63a41b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f639c037040 tx=0x7f639c035d10 comp rx=0 tx=0).stop 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 shutdown_connections 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63a410d9c0 0x7f63a41b6900 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f638c0a2ad0 0x7f638c0a4f90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63a410b010 0x7f63a4115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 --2- 172.21.15.145:0/3941196821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a41086a0 0x7f63a4114bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 >> 172.21.15.145:0/3941196821 conn(0x7f63a406e6e0 msgr2=0x7f63a4109350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 shutdown_connections 2024-03-28T18:40:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:40:57.497+0000 7f63aab37640 1 -- 172.21.15.145:0/3941196821 wait complete. 2024-03-28T18:40:59.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:40:58 smithi145 ceph-mon[32151]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 651 MiB/s wr, 1.54k op/s 2024-03-28T18:40:59.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:40:58 smithi192 ceph-mon[36589]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 651 MiB/s wr, 1.54k op/s 2024-03-28T18:40:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:40:58 smithi164 ceph-mon[36248]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 651 MiB/s wr, 1.54k op/s 2024-03-28T18:41:01.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:00 smithi145 ceph-mon[32151]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 772 MiB/s wr, 1.54k op/s 2024-03-28T18:41:01.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:00 smithi192 ceph-mon[36589]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 772 MiB/s wr, 1.54k op/s 2024-03-28T18:41:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:00 smithi164 ceph-mon[36248]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 772 MiB/s wr, 1.54k op/s 2024-03-28T18:41:03.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:02 smithi192 ceph-mon[36589]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 681 MiB/s wr, 1.36k op/s 2024-03-28T18:41:03.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:02 smithi164 ceph-mon[36248]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 681 MiB/s wr, 1.36k op/s 2024-03-28T18:41:03.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:02 smithi145 ceph-mon[32151]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 681 MiB/s wr, 1.36k op/s 2024-03-28T18:41:05.080 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:04 smithi145 ceph-mon[32151]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.33k op/s 2024-03-28T18:41:05.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:04 smithi192 ceph-mon[36589]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.33k op/s 2024-03-28T18:41:05.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:04 smithi164 ceph-mon[36248]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 684 MiB/s wr, 1.33k op/s 2024-03-28T18:41:07.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:06 smithi192 ceph-mon[36589]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 833 MiB/s wr, 1.60k op/s 2024-03-28T18:41:07.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:06 smithi164 ceph-mon[36248]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 833 MiB/s wr, 1.60k op/s 2024-03-28T18:41:07.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:06 smithi145 ceph-mon[32151]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 833 MiB/s wr, 1.60k op/s 2024-03-28T18:41:09.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:08 smithi192 ceph-mon[36589]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.08k op/s 2024-03-28T18:41:09.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:08 smithi164 ceph-mon[36248]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.08k op/s 2024-03-28T18:41:09.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:08 smithi145 ceph-mon[32151]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 679 MiB/s wr, 1.08k op/s 2024-03-28T18:41:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.747+0000 7fc2c359e640 1 -- 172.21.15.145:0/2888555855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c4072950 msgr2=0x7fc2c4072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.747+0000 7fc2c359e640 1 --2- 172.21.15.145:0/2888555855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c4072950 0x7fc2c4072d70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009900 tx=0x7fc2b402bc60 comp rx=0 tx=0).stop 2024-03-28T18:41:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.751+0000 7fc2c359e640 1 -- 172.21.15.145:0/2888555855 shutdown_connections 2024-03-28T18:41:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.751+0000 7fc2c359e640 1 --2- 172.21.15.145:0/2888555855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.751+0000 7fc2c359e640 1 --2- 172.21.15.145:0/2888555855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40732b0 0x7fc2c4108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.751+0000 7fc2c359e640 1 --2- 172.21.15.145:0/2888555855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c4072950 0x7fc2c4072d70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.751+0000 7fc2c359e640 1 -- 172.21.15.145:0/2888555855 >> 172.21.15.145:0/2888555855 conn(0x7fc2c406e510 msgr2=0x7fc2c4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.759+0000 7fc2c359e640 1 -- 172.21.15.145:0/2888555855 shutdown_connections 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.759+0000 7fc2c359e640 1 -- 172.21.15.145:0/2888555855 wait complete. 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.760+0000 7fc2c359e640 1 Processor -- start 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 -- start start 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c4072950 0x7fc2c4119910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c40732b0 0x7fc2c4119e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4110820 con 0x7fc2c40732b0 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41106a0 con 0x7fc2c4109390 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c359e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41109a0 con 0x7fc2c4072950 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57426/0 (socket says 172.21.15.145:57426) 2024-03-28T18:41:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.761+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/3163680203 learned_addr learned my addr 172.21.15.145:0/3163680203 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.762+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c40732b0 0x7fc2c4119e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.762+0000 7fc2c259c640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c4072950 0x7fc2c4119910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.778 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.763+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/3163680203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c4072950 msgr2=0x7fc2c4119910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.763+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c4072950 0x7fc2c4119910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.763+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c40732b0 msgr2=0x7fc2c4119e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.763+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c40732b0 0x7fc2c4119e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.763+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/3163680203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c41186d0 con 0x7fc2c4109390 2024-03-28T18:41:10.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.770+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00eaa0 tx=0x7fc2bc00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.774+0000 7fc2bb7fe640 1 -- 172.21.15.145:0/3163680203 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00cdb0 con 0x7fc2c4109390 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.774+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c41189c0 con 0x7fc2c4109390 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.774+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c4079900 con 0x7fc2c4109390 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.774+0000 7fc2bb7fe640 1 -- 172.21.15.145:0/3163680203 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc005020 con 0x7fc2c4109390 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.774+0000 7fc2bb7fe640 1 -- 172.21.15.145:0/3163680203 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0116b0 con 0x7fc2c4109390 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.778+0000 7fc2cb59a740 1 Processor -- start 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 -- start start 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x5627630d5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562763034b10 con 0x5627630d0510 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x562762fdd900 con 0x5627630d5680 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.783+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562763030bb0 con 0x5627630cfb20 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.784+0000 7fc2bbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.784+0000 7fc2bbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:42172/0 (socket says 172.21.15.145:42172) 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.784+0000 7fc2bbfff640 1 -- 172.21.15.145:0/2428102964 learned_addr learned my addr 172.21.15.145:0/2428102964 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:10.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.784+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 -- 172.21.15.145:0/2428102964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 msgr2=0x5627630d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 -- 172.21.15.145:0/2428102964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 msgr2=0x5627630d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x5627630d5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 -- 172.21.15.145:0/2428102964 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562762d49210 con 0x5627630cfb20 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.785+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac017730 tx=0x7fc2ac0094c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0839d03df683800 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.786+0000 7fc2bb7fe640 1 -- 172.21.15.145:0/3163680203 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc2bc004a40 con 0x7fc2c4109390 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.787+0000 7fc2b8ff9640 1 -- 172.21.15.145:0/2428102964 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac015760 con 0x5627630cfb20 2024-03-28T18:41:10.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.787+0000 7fc2b8ff9640 1 -- 172.21.15.145:0/2428102964 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac01d070 con 0x5627630cfb20 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2428102964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 msgr2=0x5627630cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac017730 tx=0x7fc2ac0094c0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2428102964 shutdown_connections 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x5627630d5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2428102964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630cff40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.788+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2428102964 >> 172.21.15.145:0/2428102964 conn(0x5627630cb090 msgr2=0x5627630cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:10.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.792+0000 7fc2bb7fe640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 0x7fc2b00a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.792+0000 7fc2bb7fe640 1 -- 172.21.15.145:0/3163680203 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc2bc0cf100 con 0x7fc2c4109390 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.797+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2428102964 shutdown_connections 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.797+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2428102964 wait complete. 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.797+0000 7fc2cb59a740 1 -- start start 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.797+0000 7fc2cb59a740 1 Processor -- start 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 0x5627630fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d0510 0x5627630fc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 0x562763103340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627630fa630 con 0x5627630cfb20 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x562762ff0d50 con 0x5627630d0510 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627630fa7b0 con 0x5627630d5680 2024-03-28T18:41:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2cb59a740 10 client.0 _ll_register_callbacks cb 0x562762ff0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 0x5627630fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 0x5627630fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54606/0 (socket says 172.21.15.145:54606) 2024-03-28T18:41:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.798+0000 7fc2c259c640 1 -- 172.21.15.145:0/2366216322 learned_addr learned my addr 172.21.15.145:0/2366216322 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.803+0000 7fc2c259c640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 0x7fc2b00a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.806+0000 7fc2bbfff640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d0510 0x5627630fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.806+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 0x562763103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.807+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2366216322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d0510 msgr2=0x5627630fc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.807+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d0510 0x5627630fc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.807+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2366216322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 msgr2=0x5627630fbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.807+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 0x5627630fbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.807+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2366216322 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562762e15620 con 0x5627630d5680 2024-03-28T18:41:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.808+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 0x562763103340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009270 tx=0x7fc2bc099c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b22d9c85d84a94b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.808+0000 7fc2a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:41:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.809+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0b4020 con 0x5627630d5680 2024-03-28T18:41:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.809+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2ac01d070 con 0x5627630d5680 2024-03-28T18:41:10.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.809+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc2bc09ab60 con 0x5627630d5680 2024-03-28T18:41:10.810 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.810+0000 7fc2c259c640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 0x7fc2b00a4d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40096c0 tx=0x7fc2b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.810+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2bc09ad60 con 0x5627630d5680 2024-03-28T18:41:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.810+0000 7fc2cb59a740 10 client.25570 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.811+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562763106720 con 0x5627630d5680 2024-03-28T18:41:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac01d070 con 0x5627630d5680 2024-03-28T18:41:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc2bc00f040 con 0x5627630d5680 2024-03-28T18:41:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 10 client.25570 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 10 client.25570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 20 client.25570 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5627631082a0 0x56276310a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 4 client.25570 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:41:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.813+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562763042050 con 0x5627631082a0 2024-03-28T18:41:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.816+0000 7fc2c259c640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5627631082a0 0x56276310a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.823+0000 7fc2c259c640 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5627631082a0 0x56276310a6e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=353b6b84cb32dd59 server_cookie=cb2c13e948f15d3e in_seq=0 out_seq=0 2024-03-28T18:41:10.825 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.823+0000 7fc2a9ffb640 10 client.25570 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.842+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2366216322 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562763042050 con 0x5627631082a0 2024-03-28T18:41:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.842+0000 7fc2a9ffb640 10 client.25570 handle_command_reply: tid=0 2024-03-28T18:41:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.844+0000 7fc2cb59a740 1 client.25570 shutdown 2024-03-28T18:41:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.845+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 msgr2=0x562763103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.845+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 0x562763103340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009270 tx=0x7fc2bc099c20 comp rx=0 tx=0).stop 2024-03-28T18:41:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 shutdown_connections 2024-03-28T18:41:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5627631082a0 0x56276310a6e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630d5680 0x562763103340 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d0510 0x5627630fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2366216322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630cfb20 0x5627630fbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.847+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 >> 172.21.15.145:0/2366216322 conn(0x5627630cb090 msgr2=0x5627630ca290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.848+0000 7fc2a9ffb640 0 client.25570 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.849+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 shutdown_connections 2024-03-28T18:41:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.853+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2366216322 wait complete. 2024-03-28T18:41:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.853+0000 7fc2cb59a740 20 client.25570 trim_cache size 0 max 16384 2024-03-28T18:41:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.909+0000 7fc2cb59a740 1 Processor -- start 2024-03-28T18:41:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.909+0000 7fc2cb59a740 1 -- start start 2024-03-28T18:41:10.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.910+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.910+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763103300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.910+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.911+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a400fb80 con 0x5627630d0510 2024-03-28T18:41:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.911+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4011500 con 0x5627630d5680 2024-03-28T18:41:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.911+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.911+0000 7fc2c259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630d4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:42182/0 (socket says 172.21.15.145:42182) 2024-03-28T18:41:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.911+0000 7fc2c259c640 1 -- 172.21.15.145:0/72749575 learned_addr learned my addr 172.21.15.145:0/72749575 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.912+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.912+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a40101e0 con 0x5627630cfb20 2024-03-28T18:41:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.917+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/72749575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 msgr2=0x5627630d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.917+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630d4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.917+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/72749575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 msgr2=0x562763103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.917+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.917+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/72749575 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627630d4a10 con 0x5627630d5680 2024-03-28T18:41:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.918+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d1790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc09b410 tx=0x7fc2bc009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752f54ba980baddb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.922+0000 7fc2abfff640 1 -- 172.21.15.145:0/72749575 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0b4040 con 0x5627630d5680 2024-03-28T18:41:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.922+0000 7fc2abfff640 1 -- 172.21.15.145:0/72749575 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc004d30 con 0x5627630d5680 2024-03-28T18:41:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.923+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 msgr2=0x5627630d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.923+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d1790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc09b410 tx=0x7fc2bc009710 comp rx=0 tx=0).stop 2024-03-28T18:41:10.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.927+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 shutdown_connections 2024-03-28T18:41:10.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.927+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x5627630d1790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.927+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.927+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/72749575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x5627630d4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.927+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 >> 172.21.15.145:0/72749575 conn(0x5627630cb090 msgr2=0x5627630cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.928+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 shutdown_connections 2024-03-28T18:41:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.930+0000 7fc2cb59a740 1 -- 172.21.15.145:0/72749575 wait complete. 2024-03-28T18:41:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.930+0000 7fc2cb59a740 1 -- start start 2024-03-28T18:41:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.930+0000 7fc2cb59a740 1 Processor -- start 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x562763100820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4010ea0 con 0x5627630d0510 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4010840 con 0x5627630d5680 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a400dd40 con 0x5627630cfb20 2024-03-28T18:41:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2cb59a740 10 client.0 _ll_register_callbacks cb 0x7fc2bc09bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2c2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2c2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57460/0 (socket says 172.21.15.145:57460) 2024-03-28T18:41:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.931+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2699341499 learned_addr learned my addr 172.21.15.145:0/2699341499 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.932+0000 7fc2c259c640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x562763100820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.933+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 msgr2=0x562763100820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.933+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x562763100820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.933+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2699341499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 msgr2=0x562763100d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.933+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.933+0000 7fc2c2d9d640 1 -- 172.21.15.145:0/2699341499 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562763102310 con 0x5627630d5680 2024-03-28T18:41:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.935+0000 7fc2c2d9d640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc098090 tx=0x7fc2bc09b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d39c6b483622d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.935+0000 7fc2a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:41:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.935+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0b4040 con 0x5627630d5680 2024-03-28T18:41:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.936+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56276303c3a0 con 0x5627630d5680 2024-03-28T18:41:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.936+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc2bc097460 con 0x5627630d5680 2024-03-28T18:41:10.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.937+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2bc0976a0 con 0x5627630d5680 2024-03-28T18:41:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.938+0000 7fc2cb59a740 10 client.25487 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:10.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.938+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562763102310 con 0x5627630d5680 2024-03-28T18:41:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.940+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56276303c3a0 con 0x5627630d5680 2024-03-28T18:41:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.941+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc2bc09cbe0 con 0x5627630d5680 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 10 client.25487 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 10 client.25487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 20 client.25487 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562763163ef0 0x5627631662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 4 client.25487 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:41:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.942+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562762fba710 con 0x562763163ef0 2024-03-28T18:41:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.947+0000 7fc2c259c640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562763163ef0 0x5627631662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.951+0000 7fc2c259c640 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562763163ef0 0x5627631662e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28c7f5f99cac73c0 server_cookie=ae67670c53a1ef33 in_seq=0 out_seq=0 2024-03-28T18:41:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:10.952+0000 7fc2a9ffb640 10 client.25487 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:11.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:10 smithi192 ceph-mon[36589]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 MiB/s wr, 1.28k op/s 2024-03-28T18:41:11.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:10 smithi164 ceph-mon[36248]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 MiB/s wr, 1.28k op/s 2024-03-28T18:41:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:10 smithi145 ceph-mon[32151]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 MiB/s wr, 1.28k op/s 2024-03-28T18:41:13.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:12 smithi192 ceph-mon[36589]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 736 MiB/s wr, 1.27k op/s 2024-03-28T18:41:13.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:12 smithi164 ceph-mon[36248]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 736 MiB/s wr, 1.27k op/s 2024-03-28T18:41:13.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:12 smithi145 ceph-mon[32151]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 736 MiB/s wr, 1.27k op/s 2024-03-28T18:41:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:14 smithi192 ceph-mon[36589]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 728 MiB/s wr, 1.29k op/s 2024-03-28T18:41:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:14 smithi164 ceph-mon[36248]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 728 MiB/s wr, 1.29k op/s 2024-03-28T18:41:15.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:14 smithi145 ceph-mon[32151]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 728 MiB/s wr, 1.29k op/s 2024-03-28T18:41:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2a9ffb640 1 -- 172.21.15.145:0/2699341499 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562762fba710 con 0x562763163ef0 2024-03-28T18:41:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2a9ffb640 10 client.25487 handle_command_reply: tid=0 2024-03-28T18:41:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2cb59a740 1 client.25487 shutdown 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 msgr2=0x562763101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc098090 tx=0x7fc2bc09b560 comp rx=0 tx=0).stop 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 shutdown_connections 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.570+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562763163ef0 0x5627631662e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627630d5680 0x562763101d20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5627630d0510 0x562763100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 --2- 172.21.15.145:0/2699341499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5627630cfb20 0x562763100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 >> 172.21.15.145:0/2699341499 conn(0x5627630cb090 msgr2=0x5627630cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2a9ffb640 0 client.25487 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 shutdown_connections 2024-03-28T18:41:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 1 -- 172.21.15.145:0/2699341499 wait complete. 2024-03-28T18:41:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.571+0000 7fc2cb59a740 20 client.25487 trim_cache size 0 max 16384 2024-03-28T18:41:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:41:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:41:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:41:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:41:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.572+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 msgr2=0x7fc2b00a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.572+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 0x7fc2b00a4d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40096c0 tx=0x7fc2b4005fd0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.572+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 msgr2=0x7fc2c4117f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.572+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00eaa0 tx=0x7fc2bc00ef60 comp rx=0 tx=0).stop 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 shutdown_connections 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2c4109390 0x7fc2c4117f50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2c40732b0 0x7fc2c4119e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc2b00a28c0 0x7fc2b00a4d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 --2- 172.21.15.145:0/3163680203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c4072950 0x7fc2c4119910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 >> 172.21.15.145:0/3163680203 conn(0x7fc2c406e510 msgr2=0x7fc2c410c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 shutdown_connections 2024-03-28T18:41:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:16.573+0000 7fc2c359e640 1 -- 172.21.15.145:0/3163680203 wait complete. 2024-03-28T18:41:17.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:16 smithi192 ceph-mon[36589]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 914 MiB/s wr, 1.63k op/s 2024-03-28T18:41:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:16 smithi164 ceph-mon[36248]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 914 MiB/s wr, 1.63k op/s 2024-03-28T18:41:17.280 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:16 smithi145 ceph-mon[32151]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 914 MiB/s wr, 1.63k op/s 2024-03-28T18:41:19.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:18 smithi192 ceph-mon[36589]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 693 MiB/s wr, 1.17k op/s 2024-03-28T18:41:19.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:18 smithi164 ceph-mon[36248]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 693 MiB/s wr, 1.17k op/s 2024-03-28T18:41:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:18 smithi145 ceph-mon[32151]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 693 MiB/s wr, 1.17k op/s 2024-03-28T18:41:21.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:20 smithi192 ceph-mon[36589]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 838 MiB/s wr, 1.40k op/s 2024-03-28T18:41:21.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:20 smithi164 ceph-mon[36248]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 838 MiB/s wr, 1.40k op/s 2024-03-28T18:41:21.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:20 smithi145 ceph-mon[32151]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 838 MiB/s wr, 1.40k op/s 2024-03-28T18:41:23.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:22 smithi192 ceph-mon[36589]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 773 MiB/s wr, 1.29k op/s 2024-03-28T18:41:23.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:22 smithi164 ceph-mon[36248]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 773 MiB/s wr, 1.29k op/s 2024-03-28T18:41:23.281 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:22 smithi145 ceph-mon[32151]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 773 MiB/s wr, 1.29k op/s 2024-03-28T18:41:25.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:24 smithi192 ceph-mon[36589]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 671 MiB/s wr, 1.12k op/s 2024-03-28T18:41:25.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:24 smithi164 ceph-mon[36248]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 671 MiB/s wr, 1.12k op/s 2024-03-28T18:41:25.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:24 smithi145 ceph-mon[32151]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 671 MiB/s wr, 1.12k op/s 2024-03-28T18:41:26.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:25 smithi164 ceph-mon[36248]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 MiB/s wr, 1.52k op/s 2024-03-28T18:41:26.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:25 smithi145 ceph-mon[32151]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 MiB/s wr, 1.52k op/s 2024-03-28T18:41:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:25 smithi192 ceph-mon[36589]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 MiB/s wr, 1.52k op/s 2024-03-28T18:41:29.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:28 smithi145 ceph-mon[32151]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.09k op/s 2024-03-28T18:41:29.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:28 smithi192 ceph-mon[36589]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.09k op/s 2024-03-28T18:41:29.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:28 smithi164 ceph-mon[36248]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.09k op/s 2024-03-28T18:41:29.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.874+0000 7ff041c56640 1 -- 172.21.15.145:0/79371429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 msgr2=0x7ff03c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.876 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.874+0000 7ff041c56640 1 --2- 172.21.15.145:0/79371429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c072d70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff034009ca0 tx=0x7ff03402c140 comp rx=0 tx=0).stop 2024-03-28T18:41:29.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.877+0000 7ff041c56640 1 -- 172.21.15.145:0/79371429 shutdown_connections 2024-03-28T18:41:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.877+0000 7ff041c56640 1 --2- 172.21.15.145:0/79371429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff03c109390 0x7ff03c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.877+0000 7ff041c56640 1 --2- 172.21.15.145:0/79371429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff03c0732b0 0x7ff03c108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.877+0000 7ff041c56640 1 --2- 172.21.15.145:0/79371429 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c072d70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.877+0000 7ff041c56640 1 -- 172.21.15.145:0/79371429 >> 172.21.15.145:0/79371429 conn(0x7ff03c06e510 msgr2=0x7ff03c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.896+0000 7ff041c56640 1 -- 172.21.15.145:0/79371429 shutdown_connections 2024-03-28T18:41:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.902+0000 7ff041c56640 1 -- 172.21.15.145:0/79371429 wait complete. 2024-03-28T18:41:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.903+0000 7ff041c56640 1 Processor -- start 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.906+0000 7ff041c56640 1 -- start start 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.906+0000 7ff041c56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.907+0000 7ff041c56640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff03c0732b0 0x7ff03c1a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.907+0000 7ff041c56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff03c109390 0x7ff03c1ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.907+0000 7ff041c56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c11c0d0 con 0x7ff03c072950 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.907+0000 7ff041c56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c11bf50 con 0x7ff03c109390 2024-03-28T18:41:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.907+0000 7ff041c56640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c11c250 con 0x7ff03c0732b0 2024-03-28T18:41:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.910+0000 7ff040c54640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.910+0000 7ff040c54640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53376/0 (socket says 172.21.15.145:53376) 2024-03-28T18:41:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.910+0000 7ff040c54640 1 -- 172.21.15.145:0/3541186098 learned_addr learned my addr 172.21.15.145:0/3541186098 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 -- 172.21.15.145:0/3541186098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff03c0732b0 msgr2=0x7ff03c1a6e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff03c0732b0 0x7ff03c1a6e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 -- 172.21.15.145:0/3541186098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff03c109390 msgr2=0x7ff03c1ade40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff03c109390 0x7ff03c1ade40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 -- 172.21.15.145:0/3541186098 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c10ff00 con 0x7ff03c072950 2024-03-28T18:41:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.911+0000 7ff040c54640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff034006010 tx=0x7ff034003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.912+0000 7ff039ffb640 1 -- 172.21.15.145:0/3541186098 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034041020 con 0x7ff03c072950 2024-03-28T18:41:29.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.912+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03c110190 con 0x7ff03c072950 2024-03-28T18:41:29.912 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.912+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03c110700 con 0x7ff03c072950 2024-03-28T18:41:29.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.913+0000 7ff039ffb640 1 -- 172.21.15.145:0/3541186098 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff034004720 con 0x7ff03c072950 2024-03-28T18:41:29.913 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.913+0000 7ff039ffb640 1 -- 172.21.15.145:0/3541186098 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034039040 con 0x7ff03c072950 2024-03-28T18:41:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.918+0000 7ff045b78740 1 Processor -- start 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 -- start start 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff039ffb640 1 -- 172.21.15.145:0/3541186098 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff034044020 con 0x7ff03c072950 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac99b20 0x561e6ac99f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6aca1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6abfeb10 con 0x561e6ac99b20 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6aba7900 con 0x561e6ac9f680 2024-03-28T18:41:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.920+0000 7ff045b78740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6abfabb0 con 0x561e6ac9a510 2024-03-28T18:41:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.921+0000 7ff040c54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.921+0000 7ff040c54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:44592/0 (socket says 172.21.15.145:44592) 2024-03-28T18:41:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.921+0000 7ff040c54640 1 -- 172.21.15.145:0/940170829 learned_addr learned my addr 172.21.15.145:0/940170829 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.921+0000 7ff040c54640 1 -- 172.21.15.145:0/940170829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 msgr2=0x561e6aca1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:41:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff040c54640 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6aca1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff040c54640 1 -- 172.21.15.145:0/940170829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac99b20 msgr2=0x561e6ac99f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff040c54640 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac99b20 0x561e6ac99f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff040c54640 1 -- 172.21.15.145:0/940170829 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e6a913210 con 0x561e6ac9a510 2024-03-28T18:41:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff039ffb640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 0x7ff0280a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.922+0000 7ff040c54640 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff0340f4f90 tx=0x7ff0340f91a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c09aa046cee1dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.923+0000 7ff039ffb640 1 -- 172.21.15.145:0/3541186098 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff0340f1080 con 0x7ff03c072950 2024-03-28T18:41:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.924+0000 7ff032ffd640 1 -- 172.21.15.145:0/940170829 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340b98b0 con 0x561e6ac9a510 2024-03-28T18:41:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.924+0000 7ff032ffd640 1 -- 172.21.15.145:0/940170829 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0340fe030 con 0x561e6ac9a510 2024-03-28T18:41:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.924+0000 7ff032ffd640 1 -- 172.21.15.145:0/940170829 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340bcf70 con 0x561e6ac9a510 2024-03-28T18:41:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.925+0000 7ff045b78740 1 -- 172.21.15.145:0/940170829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 msgr2=0x561e6ac9f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.925+0000 7ff045b78740 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff0340f4f90 tx=0x7ff0340f91a0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.925+0000 7ff041455640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 0x7ff0280a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.928+0000 7ff045b78740 1 -- 172.21.15.145:0/940170829 shutdown_connections 2024-03-28T18:41:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.928+0000 7ff045b78740 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6aca1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.928+0000 7ff045b78740 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac9a510 0x561e6ac9f010 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.928+0000 7ff045b78740 1 --2- 172.21.15.145:0/940170829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac99b20 0x561e6ac99f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.928+0000 7ff045b78740 1 -- 172.21.15.145:0/940170829 >> 172.21.15.145:0/940170829 conn(0x561e6ac95090 msgr2=0x561e6ac95490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.931+0000 7ff041455640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 0x7ff0280a4e40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff024006c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.939+0000 7ff045b78740 1 -- 172.21.15.145:0/940170829 shutdown_connections 2024-03-28T18:41:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.945+0000 7ff045b78740 1 -- 172.21.15.145:0/940170829 wait complete. 2024-03-28T18:41:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.945+0000 7ff045b78740 1 -- start start 2024-03-28T18:41:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.945+0000 7ff045b78740 1 Processor -- start 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6aca23f0 con 0x561e6ac9a510 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6abbad50 con 0x561e6ac9f680 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e6aca2570 con 0x561e6ac99b20 2024-03-28T18:41:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.947+0000 7ff045b78740 10 client.0 _ll_register_callbacks cb 0x561e6abba720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.951+0000 7ff041455640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.951+0000 7ff040c54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.951+0000 7ff03bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.951+0000 7ff03bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53396/0 (socket says 172.21.15.145:53396) 2024-03-28T18:41:29.952 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.951+0000 7ff03bfff640 1 -- 172.21.15.145:0/4027990492 learned_addr learned my addr 172.21.15.145:0/4027990492 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.958+0000 7ff03bfff640 1 -- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 msgr2=0x561e6aca3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.958+0000 7ff03bfff640 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.958+0000 7ff03bfff640 1 -- 172.21.15.145:0/4027990492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 msgr2=0x561e6acab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:29.958 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.958+0000 7ff03bfff640 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.958+0000 7ff03bfff640 1 -- 172.21.15.145:0/4027990492 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e6a9df620 con 0x561e6ac9a510 2024-03-28T18:41:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.960+0000 7ff03bfff640 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009d80 tx=0x7ff02c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a9cd3a451c066d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.960+0000 7ff01bfff640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:41:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.962+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c011420 con 0x561e6ac9a510 2024-03-28T18:41:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.962+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0340fe030 con 0x561e6ac9a510 2024-03-28T18:41:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.962+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff02c0115c0 con 0x561e6ac9a510 2024-03-28T18:41:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.963+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff02c011800 con 0x561e6ac9a510 2024-03-28T18:41:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.963+0000 7ff045b78740 10 client.15732 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.963+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561e6acae5b0 con 0x561e6ac9a510 2024-03-28T18:41:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.963+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0340fe030 con 0x561e6ac9a510 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff02c01bcc0 con 0x561e6ac9a510 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 10 client.15732 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 10 client.15732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 20 client.15732 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6acb0110 0x561e6acb2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 4 client.15732 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:41:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.964+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e6ac0c050 con 0x561e6acb0110 2024-03-28T18:41:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.966+0000 7ff040c54640 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6acb0110 0x561e6acb2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.976+0000 7ff040c54640 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6acb0110 0x561e6acb2590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82e11a2d05a4c3b server_cookie=af78c39c850461eb in_seq=0 out_seq=0 2024-03-28T18:41:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.976+0000 7ff01bfff640 10 client.15732 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.980+0000 7ff01bfff640 1 -- 172.21.15.145:0/4027990492 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e6ac0c050 con 0x561e6acb0110 2024-03-28T18:41:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.980+0000 7ff01bfff640 10 client.15732 handle_command_reply: tid=0 2024-03-28T18:41:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.980+0000 7ff045b78740 1 client.15732 shutdown 2024-03-28T18:41:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.989+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 msgr2=0x561e6aca4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.989+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009d80 tx=0x7ff02c00b4b0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 shutdown_connections 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6acb0110 0x561e6acb2590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9a510 0x561e6aca4100 secure :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009d80 tx=0x7ff02c00b4b0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 --2- 172.21.15.145:0/4027990492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.991+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 >> 172.21.15.145:0/4027990492 conn(0x561e6ac95090 msgr2=0x561e6ac94290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.992+0000 7ff01bfff640 0 client.15732 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.992+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 shutdown_connections 2024-03-28T18:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.998+0000 7ff045b78740 1 -- 172.21.15.145:0/4027990492 wait complete. 2024-03-28T18:41:29.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:29.999+0000 7ff045b78740 20 client.15732 trim_cache size 0 max 16384 2024-03-28T18:41:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.036+0000 7ff045b78740 1 Processor -- start 2024-03-28T18:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 -- start start 2024-03-28T18:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac958d0 0x561e6aca33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c00fb40 con 0x561e6ac958d0 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0114c0 con 0x561e6ac9f680 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.037+0000 7ff045b78740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0101a0 con 0x561e6ac99b20 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.038+0000 7ff040c54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.038+0000 7ff040c54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:44616/0 (socket says 172.21.15.145:44616) 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.038+0000 7ff040c54640 1 -- 172.21.15.145:0/4047759073 learned_addr learned my addr 172.21.15.145:0/4047759073 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.039+0000 7ff041455640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac958d0 0x561e6aca33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.039+0000 7ff03bfff640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff040c54640 1 -- 172.21.15.145:0/4047759073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 msgr2=0x561e6acab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff040c54640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff040c54640 1 -- 172.21.15.145:0/4047759073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac958d0 msgr2=0x561e6aca33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff040c54640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac958d0 0x561e6aca33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff040c54640 1 -- 172.21.15.145:0/4047759073 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e6ac9b320 con 0x561e6ac99b20 2024-03-28T18:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.040+0000 7ff03bfff640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.042+0000 7ff040c54640 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b9fe0 tx=0x7ff0340f4ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7543f148aa1d98fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.042+0000 7ff0327fc640 1 -- 172.21.15.145:0/4047759073 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340bd150 con 0x561e6ac99b20 2024-03-28T18:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.042+0000 7ff0327fc640 1 -- 172.21.15.145:0/4047759073 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0340fe040 con 0x561e6ac99b20 2024-03-28T18:41:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.042+0000 7ff0327fc640 1 -- 172.21.15.145:0/4047759073 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340bd3e0 con 0x561e6ac99b20 2024-03-28T18:41:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.044+0000 7ff045b78740 1 -- 172.21.15.145:0/4047759073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 msgr2=0x561e6aca2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.044+0000 7ff045b78740 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b9fe0 tx=0x7ff0340f4ac0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.048+0000 7ff045b78740 1 -- 172.21.15.145:0/4047759073 shutdown_connections 2024-03-28T18:41:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.048+0000 7ff045b78740 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac958d0 0x561e6aca33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.048+0000 7ff045b78740 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac9f680 0x561e6acab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.048+0000 7ff045b78740 1 --2- 172.21.15.145:0/4047759073 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac99b20 0x561e6aca2eb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.048+0000 7ff045b78740 1 -- 172.21.15.145:0/4047759073 >> 172.21.15.145:0/4047759073 conn(0x561e6ac95090 msgr2=0x561e6ac982f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.053+0000 7ff045b78740 1 -- 172.21.15.145:0/4047759073 shutdown_connections 2024-03-28T18:41:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.056+0000 7ff045b78740 1 -- 172.21.15.145:0/4047759073 wait complete. 2024-03-28T18:41:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.056+0000 7ff045b78740 1 -- start start 2024-03-28T18:41:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.056+0000 7ff045b78740 1 Processor -- start 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac958d0 0x561e6aca88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 0x561e6aca8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 0x561e6aca9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c010e60 con 0x561e6ac9f680 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c010800 con 0x561e6ac99b20 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c00dd00 con 0x561e6ac958d0 2024-03-28T18:41:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff045b78740 10 client.0 _ll_register_callbacks cb 0x7ff02c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff041455640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 0x561e6aca9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff041455640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 0x561e6aca9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53406/0 (socket says 172.21.15.145:53406) 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.057+0000 7ff041455640 1 -- 172.21.15.145:0/928462124 learned_addr learned my addr 172.21.15.145:0/928462124 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 0x561e6aca8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 -- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac958d0 msgr2=0x561e6aca88e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac958d0 0x561e6aca88e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 -- 172.21.15.145:0/928462124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 msgr2=0x561e6aca9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 0x561e6aca9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.058+0000 7ff03bfff640 1 -- 172.21.15.145:0/928462124 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e6acaa420 con 0x561e6ac99b20 2024-03-28T18:41:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.059+0000 7ff03bfff640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 0x561e6aca8e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff02c008000 tx=0x7ff02c00bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b347e4bb9cd6d5c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.059+0000 7ff01bfff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:41:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.059+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c02a020 con 0x561e6ac99b20 2024-03-28T18:41:30.059 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.059+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff02c011ba0 con 0x561e6ac99b20 2024-03-28T18:41:30.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.062+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0340fe040 con 0x561e6ac99b20 2024-03-28T18:41:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.062+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff02c011630 con 0x561e6ac99b20 2024-03-28T18:41:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.063+0000 7ff045b78740 10 client.25496 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:30.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.063+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561e6ad0bbc0 con 0x561e6ac99b20 2024-03-28T18:41:30.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0340fe040 con 0x561e6ac99b20 2024-03-28T18:41:30.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff02c023070 con 0x561e6ac99b20 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 10 client.25496 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 10 client.25496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 20 client.25496 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6ad0c4c0 0x561e6ad0e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 4 client.25496 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561e6ac063a0 con 0x561e6ad0c4c0 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.064+0000 7ff040c54640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6ad0c4c0 0x561e6ad0e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.066+0000 7ff040c54640 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6ad0c4c0 0x561e6ad0e8e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df26781530952c1a server_cookie=e05fe2b3ecbd1aac in_seq=0 out_seq=0 2024-03-28T18:41:30.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:30.066+0000 7ff01bfff640 10 client.25496 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:31.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:30 smithi145 ceph-mon[32151]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 788 MiB/s wr, 1.28k op/s 2024-03-28T18:41:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:30 smithi192 ceph-mon[36589]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 788 MiB/s wr, 1.28k op/s 2024-03-28T18:41:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:30 smithi164 ceph-mon[36248]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 788 MiB/s wr, 1.28k op/s 2024-03-28T18:41:32.029 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:31 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:32.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:31 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:32.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:31 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff01bfff640 1 -- 172.21.15.145:0/928462124 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561e6ac063a0 con 0x561e6ad0c4c0 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff01bfff640 10 client.25496 handle_command_reply: tid=0 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 client.25496 shutdown 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 msgr2=0x561e6aca8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 0x561e6aca8e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff02c008000 tx=0x7ff02c00bd70 comp rx=0 tx=0).stop 2024-03-28T18:41:32.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 shutdown_connections 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x561e6ad0c4c0 0x561e6ad0e8e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561e6ac9f680 0x561e6aca9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561e6ac99b20 0x561e6aca8e20 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff02c008000 tx=0x7ff02c00bd70 comp rx=0 tx=0).stop 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 --2- 172.21.15.145:0/928462124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561e6ac958d0 0x561e6aca88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 >> 172.21.15.145:0/928462124 conn(0x561e6ac95090 msgr2=0x561e6aca2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.727+0000 7ff01bfff640 0 client.25496 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.728+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 shutdown_connections 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.728+0000 7ff045b78740 1 -- 172.21.15.145:0/928462124 wait complete. 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.728+0000 7ff045b78740 20 client.25496 trim_cache size 0 max 16384 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:41:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:41:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:41:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:41:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.729+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 msgr2=0x7ff0280a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.729+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 0x7ff0280a4e40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff024006c20 comp rx=0 tx=0).stop 2024-03-28T18:41:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.729+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 msgr2=0x7ff03c1a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.729+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff034006010 tx=0x7ff034003f10 comp rx=0 tx=0).stop 2024-03-28T18:41:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 shutdown_connections 2024-03-28T18:41:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff03c109390 0x7ff03c1ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff0280a2980 0x7ff0280a4e40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff03c0732b0 0x7ff03c1a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 --2- 172.21.15.145:0/3541186098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff03c072950 0x7ff03c1a68e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.734+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 >> 172.21.15.145:0/3541186098 conn(0x7ff03c06e510 msgr2=0x7ff03c070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.737+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 shutdown_connections 2024-03-28T18:41:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:32.739+0000 7ff041c56640 1 -- 172.21.15.145:0/3541186098 wait complete. 2024-03-28T18:41:33.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:32 smithi145 ceph-mon[32151]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 805 MiB/s wr, 1.34k op/s 2024-03-28T18:41:33.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:33.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:33.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:41:33.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-mon[36589]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 805 MiB/s wr, 1.34k op/s 2024-03-28T18:41:33.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:33.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:33.086 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f67b78f3700 time 2024-03-28T18:41:32.740268+0000 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67c9f2d4a1] 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: clone() 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: *** Caught signal (Aborted) ** 2024-03-28T18:41:33.087 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: in thread 7f67b78f3700 thread_name:md_log_replay 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67c8cb4d20] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: gsignal() 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: abort() 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67c9f2d4fb] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 7: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 9: clone() 2024-03-28T18:41:33.088 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:32.740+0000 7f67b78f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f67b78f3700 time 2024-03-28T18:41:32.740268+0000 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67c9f2d4a1] 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: clone() 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:32.741+0000 7f67b78f3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: in thread 7f67b78f3700 thread_name:md_log_replay 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.089 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67c8cb4d20] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: gsignal() 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: abort() 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67c9f2d4fb] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 7: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 9: clone() 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:41:33.090 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: -1> 2024-03-28T18:41:32.740+0000 7f67b78f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f67b78f3700 time 2024-03-28T18:41:32.740268+0000 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67c9f2d4a1] 2024-03-28T18:41:33.091 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: clone() 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 0> 2024-03-28T18:41:32.741+0000 7f67b78f3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: in thread 7f67b78f3700 thread_name:md_log_replay 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.092 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67c8cb4d20] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: gsignal() 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: abort() 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67c9f2d4fb] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 7: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 9: clone() 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:41:33.093 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: -1> 2024-03-28T18:41:32.740+0000 7f67b78f3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f67b78f3700 time 2024-03-28T18:41:32.740268+0000 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f67c9f2d4a1] 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: clone() 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 0> 2024-03-28T18:41:32.741+0000 7f67b78f3700 -1 *** Caught signal (Aborted) ** 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: in thread 7f67b78f3700 thread_name:md_log_replay 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.094 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f67c8cb4d20] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2: gsignal() 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 3: abort() 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f67c9f2d4fb] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f67c9f2d667] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 6: (MDLog::_replay_thread()+0xd8) [0x557a9869ec18] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 7: (MDLog::ReplayThread::entry()+0x11) [0x557a986a5771] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 8: /lib64/libpthread.so.0(+0x81ca) [0x7f67c8caa1ca] 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 9: clone() 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[66250]: 2024-03-28T18:41:33.095 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:32 smithi192 podman[69346]: 2024-03-28 18:41:32.885727521 +0000 UTC m=+0.058758572 container died 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T18:41:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:32 smithi164 ceph-mon[36248]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 805 MiB/s wr, 1.34k op/s 2024-03-28T18:41:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:41:33.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:33 smithi192 podman[69346]: 2024-03-28 18:41:33.558085489 +0000 UTC m=+0.731116537 container remove 2d59c9eb3b676bd071e8ee6d994e2c1c073bcf4932cfeea85a2e7af93c25c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T18:41:33.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:33 smithi192 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-28T18:41:34.408 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:34 smithi192 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-28T18:41:34.408 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:34 smithi192 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service: Consumed 17.020s CPU time. 2024-03-28T18:41:35.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:34 smithi145 ceph-mon[32151]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 671 MiB/s wr, 1.14k op/s 2024-03-28T18:41:35.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:34 smithi192 ceph-mon[36589]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 671 MiB/s wr, 1.14k op/s 2024-03-28T18:41:35.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:34 smithi164 ceph-mon[36248]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 671 MiB/s wr, 1.14k op/s 2024-03-28T18:41:37.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:36 smithi145 ceph-mon[32151]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 917 MiB/s wr, 1.55k op/s 2024-03-28T18:41:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:36 smithi192 ceph-mon[36589]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 917 MiB/s wr, 1.55k op/s 2024-03-28T18:41:37.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:36 smithi164 ceph-mon[36248]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 917 MiB/s wr, 1.55k op/s 2024-03-28T18:41:39.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:38 smithi145 ceph-mon[32151]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 656 MiB/s wr, 1.14k op/s 2024-03-28T18:41:39.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:38 smithi192 ceph-mon[36589]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 656 MiB/s wr, 1.14k op/s 2024-03-28T18:41:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:38 smithi164 ceph-mon[36248]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 656 MiB/s wr, 1.14k op/s 2024-03-28T18:41:41.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:40 smithi145 ceph-mon[32151]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 777 MiB/s wr, 1.34k op/s 2024-03-28T18:41:41.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:40 smithi192 ceph-mon[36589]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 777 MiB/s wr, 1.34k op/s 2024-03-28T18:41:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:40 smithi164 ceph-mon[36248]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 777 MiB/s wr, 1.34k op/s 2024-03-28T18:41:43.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:42 smithi145 ceph-mon[32151]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 801 MiB/s wr, 1.36k op/s 2024-03-28T18:41:43.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:42 smithi192 ceph-mon[36589]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 801 MiB/s wr, 1.36k op/s 2024-03-28T18:41:43.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:42 smithi164 ceph-mon[36248]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 801 MiB/s wr, 1.36k op/s 2024-03-28T18:41:44.435 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:44 smithi192 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-28T18:41:44.435 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:44 smithi192 systemd[1]: Stopped Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:41:44.435 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:44 smithi192 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service: Consumed 17.020s CPU time. 2024-03-28T18:41:44.435 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:44 smithi192 systemd[1]: Starting Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:41:44.776 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:44 smithi192 podman[69441]: 2024-03-28 18:41:44.346879706 +0000 UTC m=+0.037668085 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:41:45.030 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:44 smithi145 ceph-mon[32151]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 657 MiB/s wr, 1.08k op/s 2024-03-28T18:41:45.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:44 smithi192 ceph-mon[36589]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 657 MiB/s wr, 1.08k op/s 2024-03-28T18:41:45.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:44 smithi164 ceph-mon[36248]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 657 MiB/s wr, 1.08k op/s 2024-03-28T18:41:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.040+0000 7f62f3fff640 1 -- 172.21.15.145:0/3240040758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 msgr2=0x7f62f4073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.040+0000 7f62f3fff640 1 --2- 172.21.15.145:0/3240040758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f4073060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62e800fb60 tx=0x7f62e802fcf0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.041+0000 7f62f3fff640 1 -- 172.21.15.145:0/3240040758 shutdown_connections 2024-03-28T18:41:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.041+0000 7f62f3fff640 1 --2- 172.21.15.145:0/3240040758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f410f9e0 0x7f62f4111dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.041+0000 7f62f3fff640 1 --2- 172.21.15.145:0/3240040758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f4073060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.041+0000 7f62f3fff640 1 --2- 172.21.15.145:0/3240040758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62f4109280 0x7f62f40726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.041+0000 7f62f3fff640 1 -- 172.21.15.145:0/3240040758 >> 172.21.15.145:0/3240040758 conn(0x7f62f406e510 msgr2=0x7f62f4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.042+0000 7f62f3fff640 1 -- 172.21.15.145:0/3240040758 shutdown_connections 2024-03-28T18:41:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.042+0000 7f62f3fff640 1 -- 172.21.15.145:0/3240040758 wait complete. 2024-03-28T18:41:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.043+0000 7f62f3fff640 1 Processor -- start 2024-03-28T18:41:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.043+0000 7f62f3fff640 1 -- start start 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.044+0000 7f62f3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.044+0000 7f62f3fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62f4109280 0x7f62f419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.044+0000 7f62f3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f410f9e0 0x7f62f41a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f3fff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4113820 con 0x7f62f4109280 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f3fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41136a0 con 0x7f62f410f9e0 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f3fff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41139a0 con 0x7f62f4072be0 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:34184/0 (socket says 172.21.15.145:34184) 2024-03-28T18:41:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.045+0000 7f62f2ffd640 1 -- 172.21.15.145:0/1206823842 learned_addr learned my addr 172.21.15.145:0/1206823842 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 -- 172.21.15.145:0/1206823842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f410f9e0 msgr2=0x7f62f41a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f410f9e0 0x7f62f41a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 -- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62f4109280 msgr2=0x7f62f419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62f4109280 0x7f62f419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 -- 172.21.15.145:0/1206823842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f41a5ad0 con 0x7f62f4072be0 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.046+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00eda0 tx=0x7f62ec00c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.049+0000 7f62dbfff640 1 -- 172.21.15.145:0/1206823842 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec003b70 con 0x7f62f4072be0 2024-03-28T18:41:46.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.049+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f41a5dc0 con 0x7f62f4072be0 2024-03-28T18:41:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.049+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41a6300 con 0x7f62f4072be0 2024-03-28T18:41:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.050+0000 7f62dbfff640 1 -- 172.21.15.145:0/1206823842 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62ec003d10 con 0x7f62f4072be0 2024-03-28T18:41:46.050 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.050+0000 7f62dbfff640 1 -- 172.21.15.145:0/1206823842 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec017b40 con 0x7f62f4072be0 2024-03-28T18:41:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.054+0000 7f62dbfff640 1 -- 172.21.15.145:0/1206823842 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f62ec017ce0 con 0x7f62f4072be0 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62dbfff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 0x7f62dc0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 Processor -- start 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 -- start start 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8333b20 0x5594a8333f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a8298b10 con 0x5594a8333b20 2024-03-28T18:41:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a8241900 con 0x5594a8334510 2024-03-28T18:41:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.055+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a8294bb0 con 0x5594a8339680 2024-03-28T18:41:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62dbfff640 1 -- 172.21.15.145:0/1206823842 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f62ec0cf5e0 con 0x7f62f4072be0 2024-03-28T18:41:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62f27fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62f27fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52186/0 (socket says 172.21.15.145:52186) 2024-03-28T18:41:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62f27fc640 1 -- 172.21.15.145:0/669254834 learned_addr learned my addr 172.21.15.145:0/669254834 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.056+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:34200/0 (socket says 172.21.15.145:34200) 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f27fc640 1 -- 172.21.15.145:0/669254834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 msgr2=0x5594a833bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f27fc640 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f27fc640 1 -- 172.21.15.145:0/669254834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8333b20 msgr2=0x5594a8333f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f27fc640 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8333b20 0x5594a8333f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f27fc640 1 -- 172.21.15.145:0/669254834 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594a7fad210 con 0x5594a8334510 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.057+0000 7f62f37fe640 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.058+0000 7f62f27fc640 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62e800fa80 tx=0x7f62e8005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e30ee1043f8a97c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.058+0000 7f62f27fc640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 0x7f62dc0a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.060+0000 7f62d97fa640 1 -- 172.21.15.145:0/669254834 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e803d040 con 0x5594a8334510 2024-03-28T18:41:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.061+0000 7f62d97fa640 1 -- 172.21.15.145:0/669254834 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62e8038640 con 0x5594a8334510 2024-03-28T18:41:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.062+0000 7f62fc1aa740 1 -- 172.21.15.145:0/669254834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 msgr2=0x5594a8339010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.062+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62e800fa80 tx=0x7f62e8005c90 comp rx=0 tx=0).stop 2024-03-28T18:41:46.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.064+0000 7f62fc1aa740 1 -- 172.21.15.145:0/669254834 shutdown_connections 2024-03-28T18:41:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.064+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8339680 0x5594a833bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.064+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8339010 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.064+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/669254834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8333b20 0x5594a8333f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.064+0000 7f62fc1aa740 1 -- 172.21.15.145:0/669254834 >> 172.21.15.145:0/669254834 conn(0x5594a832f090 msgr2=0x5594a832f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.068+0000 7f62fc1aa740 1 -- 172.21.15.145:0/669254834 shutdown_connections 2024-03-28T18:41:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.068+0000 7f62f27fc640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 0x7f62dc0a4e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036570 tx=0x7f62e8036dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.068+0000 7f62fc1aa740 1 -- 172.21.15.145:0/669254834 wait complete. 2024-03-28T18:41:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.068+0000 7f62fc1aa740 1 -- start start 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.068+0000 7f62fc1aa740 1 Processor -- start 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a833e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a83453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a833c680 con 0x5594a8339680 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a8254d50 con 0x5594a8334510 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594a833c800 con 0x5594a8333b20 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62fc1aa740 10 client.0 _ll_register_callbacks cb 0x5594a8254720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:34206/0 (socket says 172.21.15.145:34206) 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 -- 172.21.15.145:0/3393016738 learned_addr learned my addr 172.21.15.145:0/3393016738 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f27fc640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a833e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 -- 172.21.15.145:0/3393016738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 msgr2=0x5594a833e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a833e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 -- 172.21.15.145:0/3393016738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 msgr2=0x5594a83453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a83453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.069+0000 7f62f2ffd640 1 -- 172.21.15.145:0/3393016738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594a8079620 con 0x5594a8333b20 2024-03-28T18:41:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.074+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f62ec098540 tx=0x7f62ec09c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7b9e3d21b5309f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.077+0000 7f62d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:41:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.081+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec09bb70 con 0x5594a8333b20 2024-03-28T18:41:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.082+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62e8038640 con 0x5594a8333b20 2024-03-28T18:41:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.082+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f62ec06f4d0 con 0x5594a8333b20 2024-03-28T18:41:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.090+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62ec09fcc0 con 0x5594a8333b20 2024-03-28T18:41:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.091+0000 7f62fc1aa740 10 client.25597 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:46.092 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.091+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5594a836a7a0 con 0x5594a8333b20 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.095+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62e8038640 con 0x5594a8333b20 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f62ec00b030 con 0x5594a8333b20 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 10 client.25597 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 20 client.25597 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a836be20 0x5594a836e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 4 client.25597 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:41:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.096+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594a82a6050 con 0x5594a836be20 2024-03-28T18:41:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.171+0000 7f62f27fc640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a836be20 0x5594a836e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.178+0000 7f62f27fc640 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a836be20 0x5594a836e210 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dd414bf05ed12a4 server_cookie=f9645ce8f291e84d in_seq=0 out_seq=0 2024-03-28T18:41:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.178+0000 7f62d6ff5640 10 client.25597 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.183+0000 7f62d6ff5640 1 -- 172.21.15.145:0/3393016738 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5594a82a6050 con 0x5594a836be20 2024-03-28T18:41:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.183+0000 7f62d6ff5640 10 client.25597 handle_command_reply: tid=0 2024-03-28T18:41:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.183+0000 7f62fc1aa740 1 client.25597 shutdown 2024-03-28T18:41:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.183+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 msgr2=0x5594a833de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.183+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f62ec098540 tx=0x7f62ec09c270 comp rx=0 tx=0).stop 2024-03-28T18:41:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 shutdown_connections 2024-03-28T18:41:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a836be20 0x5594a836e210 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a83453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a833e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/3393016738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833de30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 >> 172.21.15.145:0/3393016738 conn(0x5594a832f090 msgr2=0x5594a832e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62d6ff5640 0 client.25597 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 shutdown_connections 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 1 -- 172.21.15.145:0/3393016738 wait complete. 2024-03-28T18:41:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.184+0000 7f62fc1aa740 20 client.25597 trim_cache size 0 max 16384 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.225+0000 7f62fc1aa740 1 Processor -- start 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.225+0000 7f62fc1aa740 1 -- start start 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 0x5594a8345370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 0x5594a833df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e000fa30 con 0x5594a8334510 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00113b0 con 0x5594a8339680 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0010090 con 0x5594a8333b20 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62f2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62f27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 0x5594a8345370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62f27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 0x5594a8345370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52538/0 (socket says 172.21.15.145:52538) 2024-03-28T18:41:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62f27fc640 1 -- 172.21.15.145:0/1974021336 learned_addr learned my addr 172.21.15.145:0/1974021336 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.226+0000 7f62f37fe640 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 0x5594a833df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.227+0000 7f62f37fe640 1 -- 172.21.15.145:0/1974021336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 msgr2=0x5594a833da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.227+0000 7f62f37fe640 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.227+0000 7f62f37fe640 1 -- 172.21.15.145:0/1974021336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 msgr2=0x5594a8345370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.227+0000 7f62f37fe640 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 0x5594a8345370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.227+0000 7f62f37fe640 1 -- 172.21.15.145:0/1974021336 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594a83351e0 con 0x5594a8339680 2024-03-28T18:41:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.228+0000 7f62f37fe640 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 0x5594a833df80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009840 tx=0x7f62e4009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5f7abe5b6a4974 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.231+0000 7f62d8ff9640 1 -- 172.21.15.145:0/1974021336 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4017070 con 0x5594a8339680 2024-03-28T18:41:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.231+0000 7f62d8ff9640 1 -- 172.21.15.145:0/1974021336 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62e400ee20 con 0x5594a8339680 2024-03-28T18:41:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.232+0000 7f62d8ff9640 1 -- 172.21.15.145:0/1974021336 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e40123d0 con 0x5594a8339680 2024-03-28T18:41:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.233+0000 7f62fc1aa740 1 -- 172.21.15.145:0/1974021336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 msgr2=0x5594a833df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.233+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 0x5594a833df80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009840 tx=0x7f62e4009d00 comp rx=0 tx=0).stop 2024-03-28T18:41:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.240+0000 7f62fc1aa740 1 -- 172.21.15.145:0/1974021336 shutdown_connections 2024-03-28T18:41:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.240+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8339680 0x5594a833df80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.240+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8334510 0x5594a8345370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.240+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/1974021336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a833da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.240+0000 7f62fc1aa740 1 -- 172.21.15.145:0/1974021336 >> 172.21.15.145:0/1974021336 conn(0x5594a832f090 msgr2=0x5594a832f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.245+0000 7f62fc1aa740 1 -- 172.21.15.145:0/1974021336 shutdown_connections 2024-03-28T18:41:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.245+0000 7f62fc1aa740 1 -- 172.21.15.145:0/1974021336 wait complete. 2024-03-28T18:41:46.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.245+0000 7f62fc1aa740 1 -- start start 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.245+0000 7f62fc1aa740 1 Processor -- start 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a83429c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8342f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0010d50 con 0x5594a8339680 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00106f0 con 0x5594a8334510 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e000dbf0 con 0x5594a8333b20 2024-03-28T18:41:46.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.246+0000 7f62fc1aa740 10 client.0 _ll_register_callbacks cb 0x7f62ec09b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:41:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52540/0 (socket says 172.21.15.145:52540) 2024-03-28T18:41:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 -- 172.21.15.145:0/2736044731 learned_addr learned my addr 172.21.15.145:0/2736044731 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:41:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 -- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 msgr2=0x5594a83429c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:41:46.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a83429c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 -- 172.21.15.145:0/2736044731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 msgr2=0x5594a8342f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:41:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8342f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.248 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.247+0000 7f62f37fe640 1 -- 172.21.15.145:0/2736044731 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594a8344500 con 0x5594a8339680 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.248+0000 7f62f37fe640 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f62e4000c00 tx=0x7f62e4013970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ebb1d3a622547a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.248+0000 7f62d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.248+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4029060 con 0x5594a8339680 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.249+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62e400ee20 con 0x5594a8339680 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.249+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f62e400eb40 con 0x5594a8339680 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.249+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62e4030050 con 0x5594a8339680 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.249+0000 7f62fc1aa740 10 client.15747 fetch_fsmap learned FSMap version 29 2024-03-28T18:41:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.249+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5594a83c16b0 con 0x5594a8339680 2024-03-28T18:41:46.250 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.250+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62e400ee20 con 0x5594a8339680 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f62e40245d0 con 0x5594a8339680 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 10 client.15747 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 20 client.15747 populate_metadata read hostname 'smithi145' 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a83c1ee0 0x5594a83c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 4 client.15747 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:41:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.251+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594a82a03a0 con 0x5594a83c1ee0 2024-03-28T18:41:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.253+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a83c1ee0 0x5594a83c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.263+0000 7f62d6ff5640 10 client.15747 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:46.263+0000 7f62f2ffd640 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a83c1ee0 0x5594a83c4300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee5c0181fb61bc36 server_cookie=4bea5df05fd28d7d in_seq=0 out_seq=0 2024-03-28T18:41:46.407 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:45 smithi192 podman[69441]: 2024-03-28 18:41:45.952121394 +0000 UTC m=+1.642909750 container create a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T18:41:47.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:46 smithi192 ceph-mon[36589]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 894 MiB/s wr, 1.49k op/s 2024-03-28T18:41:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:46 smithi164 ceph-mon[36248]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 894 MiB/s wr, 1.49k op/s 2024-03-28T18:41:47.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:46 smithi145 ceph-mon[32151]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 894 MiB/s wr, 1.49k op/s 2024-03-28T18:41:48.058 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:47 smithi192 ceph-mon[36589]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:41:48.058 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:47 smithi192 ceph-mon[36589]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-28T18:41:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:47 smithi164 ceph-mon[36248]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:41:48.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:47 smithi164 ceph-mon[36248]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-28T18:41:48.279 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:47 smithi145 ceph-mon[32151]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-28T18:41:48.279 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:47 smithi145 ceph-mon[32151]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-28T18:41:49.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:48 smithi192 ceph-mon[36589]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 681 MiB/s wr, 1.16k op/s 2024-03-28T18:41:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:48 smithi164 ceph-mon[36248]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 681 MiB/s wr, 1.16k op/s 2024-03-28T18:41:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:48 smithi145 ceph-mon[32151]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 681 MiB/s wr, 1.16k op/s 2024-03-28T18:41:49.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:49 smithi192 podman[69441]: 2024-03-28 18:41:49.440739594 +0000 UTC m=+5.131527963 container init a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T18:41:49.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:49 smithi192 podman[69441]: 2024-03-28 18:41:49.446752778 +0000 UTC m=+5.137541124 container start a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:41:49.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[69454]: starting mds.f at 2024-03-28T18:41:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:50 smithi192 ceph-mon[36589]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 786 MiB/s wr, 1.50k op/s 2024-03-28T18:41:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:50 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6834/2973467272,v1:172.21.15.192:6835/2973467272] up:boot 2024-03-28T18:41:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:50 smithi192 ceph-mon[36589]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:41:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:50 smithi164 ceph-mon[36248]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 786 MiB/s wr, 1.50k op/s 2024-03-28T18:41:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:50 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6834/2973467272,v1:172.21.15.192:6835/2973467272] up:boot 2024-03-28T18:41:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:50 smithi164 ceph-mon[36248]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:41:51.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:50 smithi145 ceph-mon[32151]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 786 MiB/s wr, 1.50k op/s 2024-03-28T18:41:51.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:50 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6834/2973467272,v1:172.21.15.192:6835/2973467272] up:boot 2024-03-28T18:41:51.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:50 smithi145 ceph-mon[32151]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:51.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.357+0000 7f62d6ff5640 1 -- 172.21.15.145:0/2736044731 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594a82a03a0 con 0x5594a83c1ee0 2024-03-28T18:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.357+0000 7f62d6ff5640 10 client.15747 handle_command_reply: tid=0 2024-03-28T18:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.357+0000 7f62fc1aa740 1 client.15747 shutdown 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.357+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 msgr2=0x5594a8343f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.357+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f62e4000c00 tx=0x7f62e4013970 comp rx=0 tx=0).stop 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 shutdown_connections 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594a83c1ee0 0x5594a83c4300 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594a8339680 0x5594a8343f10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594a8334510 0x5594a8342f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 --2- 172.21.15.145:0/2736044731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594a8333b20 0x5594a83429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 >> 172.21.15.145:0/2736044731 conn(0x5594a832f090 msgr2=0x5594a833d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62d6ff5640 0 client.15747 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 shutdown_connections 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 1 -- 172.21.15.145:0/2736044731 wait complete. 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.358+0000 7f62fc1aa740 20 client.15747 trim_cache size 0 max 16384 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.360+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 msgr2=0x7f62dc0a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.360+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 0x7f62dc0a4e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62e8036570 tx=0x7f62e8036dc0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.360+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 msgr2=0x7f62f419deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.360+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00eda0 tx=0x7f62ec00c6e0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 shutdown_connections 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f410f9e0 0x7f62f41a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62f4109280 0x7f62f419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f62dc0a29c0 0x7f62dc0a4e80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 --2- 172.21.15.145:0/1206823842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62f4072be0 0x7f62f419deb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 >> 172.21.15.145:0/1206823842 conn(0x7f62f406e510 msgr2=0x7f62f410b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 shutdown_connections 2024-03-28T18:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:41:51.361+0000 7f62f3fff640 1 -- 172.21.15.145:0/1206823842 wait complete. 2024-03-28T18:41:51.657 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:51 smithi192 bash[69441]: a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 2024-03-28T18:41:51.657 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:41:51 smithi192 systemd[1]: Started Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:41:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:52 smithi192 ceph-mon[36589]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 774 MiB/s wr, 1.63k op/s 2024-03-28T18:41:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:52 smithi164 ceph-mon[36248]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 774 MiB/s wr, 1.63k op/s 2024-03-28T18:41:53.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:52 smithi145 ceph-mon[32151]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 774 MiB/s wr, 1.63k op/s 2024-03-28T18:41:55.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:54 smithi192 ceph-mon[36589]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 627 MiB/s wr, 1.39k op/s 2024-03-28T18:41:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:54 smithi164 ceph-mon[36248]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 627 MiB/s wr, 1.39k op/s 2024-03-28T18:41:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:54 smithi145 ceph-mon[32151]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 627 MiB/s wr, 1.39k op/s 2024-03-28T18:41:57.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:56 smithi192 ceph-mon[36589]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 822 MiB/s wr, 1.57k op/s 2024-03-28T18:41:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:56 smithi164 ceph-mon[36248]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 822 MiB/s wr, 1.57k op/s 2024-03-28T18:41:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:56 smithi145 ceph-mon[32151]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 822 MiB/s wr, 1.57k op/s 2024-03-28T18:41:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:58 smithi164 ceph-mon[36248]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:41:59.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:58 smithi145 ceph-mon[32151]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:41:59.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:58 smithi192 ceph-mon[36589]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:42:00.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:41:59 smithi145 ceph-mon[32151]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 751 MiB/s wr, 1.51k op/s 2024-03-28T18:42:00.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:41:59 smithi192 ceph-mon[36589]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 751 MiB/s wr, 1.51k op/s 2024-03-28T18:42:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:41:59 smithi164 ceph-mon[36248]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 751 MiB/s wr, 1.51k op/s 2024-03-28T18:42:03.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:02 smithi145 ceph-mon[32151]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 681 MiB/s wr, 1.29k op/s 2024-03-28T18:42:03.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:02 smithi192 ceph-mon[36589]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 681 MiB/s wr, 1.29k op/s 2024-03-28T18:42:03.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:02 smithi164 ceph-mon[36248]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 681 MiB/s wr, 1.29k op/s 2024-03-28T18:42:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.686+0000 7f04e6151640 1 -- 172.21.15.145:0/3006886108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0109390 msgr2=0x7f04e010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.686+0000 7f04e6151640 1 --2- 172.21.15.145:0/3006886108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0109390 0x7f04e010e000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009be0 tx=0x7f04d802bf10 comp rx=0 tx=0).stop 2024-03-28T18:42:04.693 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.690+0000 7f04e6151640 1 -- 172.21.15.145:0/3006886108 shutdown_connections 2024-03-28T18:42:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.690+0000 7f04e6151640 1 --2- 172.21.15.145:0/3006886108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0109390 0x7f04e010e000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.690+0000 7f04e6151640 1 --2- 172.21.15.145:0/3006886108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04e00732b0 0x7f04e0108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.690+0000 7f04e6151640 1 --2- 172.21.15.145:0/3006886108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0072950 0x7f04e0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.690+0000 7f04e6151640 1 -- 172.21.15.145:0/3006886108 >> 172.21.15.145:0/3006886108 conn(0x7f04e006e510 msgr2=0x7f04e0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.695+0000 7f04e6151640 1 -- 172.21.15.145:0/3006886108 shutdown_connections 2024-03-28T18:42:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.696+0000 7f04e6151640 1 -- 172.21.15.145:0/3006886108 wait complete. 2024-03-28T18:42:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.696+0000 7f04e6151640 1 Processor -- start 2024-03-28T18:42:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.699+0000 7f04e6151640 1 -- start start 2024-03-28T18:42:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.699+0000 7f04e6151640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04e0072950 0x7f04e019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e6151640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e00732b0 0x7f04e019e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e6151640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e6151640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01117c0 con 0x7f04e0109390 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e6151640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0111640 con 0x7f04e00732b0 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e6151640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0111940 con 0x7f04e0072950 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e5950640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e5950640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34262/0 (socket says 172.21.15.145:34262) 2024-03-28T18:42:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.700+0000 7f04e5950640 1 -- 172.21.15.145:0/2517481155 learned_addr learned my addr 172.21.15.145:0/2517481155 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.701+0000 7f04e5950640 1 -- 172.21.15.145:0/2517481155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04e0072950 msgr2=0x7f04e019de10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:42:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.701+0000 7f04e5950640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04e0072950 0x7f04e019de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.701+0000 7f04e5950640 1 -- 172.21.15.145:0/2517481155 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e00732b0 msgr2=0x7f04e019e350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:42:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.701+0000 7f04e5950640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e00732b0 0x7f04e019e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.701+0000 7f04e5950640 1 -- 172.21.15.145:0/2517481155 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e01a5a30 con 0x7f04e0109390 2024-03-28T18:42:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.702+0000 7f04e5950640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04d8006010 tx=0x7f04d8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.703+0000 7f04d67fc640 1 -- 172.21.15.145:0/2517481155 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8043020 con 0x7f04e0109390 2024-03-28T18:42:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.703+0000 7f04d67fc640 1 -- 172.21.15.145:0/2517481155 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d80338d0 con 0x7f04e0109390 2024-03-28T18:42:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.703+0000 7f04d67fc640 1 -- 172.21.15.145:0/2517481155 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8034a20 con 0x7f04e0109390 2024-03-28T18:42:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.703+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e01a5cc0 con 0x7f04e0109390 2024-03-28T18:42:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.703+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e01a6240 con 0x7f04e0109390 2024-03-28T18:42:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.709+0000 7f04e9f98740 1 Processor -- start 2024-03-28T18:42:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.713+0000 7f04d67fc640 1 -- 172.21.15.145:0/2517481155 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f04d8046020 con 0x7f04e0109390 2024-03-28T18:42:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.713+0000 7f04e9f98740 1 -- start start 2024-03-28T18:42:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d821819f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d821821b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d82177eb10 con 0x55d821819b20 2024-03-28T18:42:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d821727900 con 0x55d82181a510 2024-03-28T18:42:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e9f98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d82177abb0 con 0x55d82181f680 2024-03-28T18:42:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e514f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e514f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42680/0 (socket says 172.21.15.145:42680) 2024-03-28T18:42:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.714+0000 7f04e514f640 1 -- 172.21.15.145:0/2505696270 learned_addr learned my addr 172.21.15.145:0/2505696270 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 -- 172.21.15.145:0/2505696270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 msgr2=0x55d821821b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d821821b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 -- 172.21.15.145:0/2505696270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 msgr2=0x55d821819f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d821819f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 -- 172.21.15.145:0/2505696270 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d821493210 con 0x55d82181a510 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.715+0000 7f04e514f640 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04d001ca90 tx=0x7f04d001cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db76fa25d2693297 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.716+0000 7f04d67fc640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 0x7f04cc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.716+0000 7f04d67fc640 1 -- 172.21.15.145:0/2517481155 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f04d80f1a30 con 0x7f04e0109390 2024-03-28T18:42:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.718+0000 7f04cb7fe640 1 -- 172.21.15.145:0/2505696270 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0027070 con 0x55d82181a510 2024-03-28T18:42:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.718+0000 7f04cb7fe640 1 -- 172.21.15.145:0/2505696270 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d0005500 con 0x55d82181a510 2024-03-28T18:42:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.718+0000 7f04cb7fe640 1 -- 172.21.15.145:0/2505696270 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0021470 con 0x55d82181a510 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 -- 172.21.15.145:0/2505696270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 msgr2=0x55d82181f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04d001ca90 tx=0x7f04d001cf50 comp rx=0 tx=0).stop 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 -- 172.21.15.145:0/2505696270 shutdown_connections 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d821821b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d82181f010 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2505696270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d821819f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.724 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.720+0000 7f04e9f98740 1 -- 172.21.15.145:0/2505696270 >> 172.21.15.145:0/2505696270 conn(0x55d821815090 msgr2=0x55d821815490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.724+0000 7f04e494e640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 0x7f04cc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.725+0000 7f04e9f98740 1 -- 172.21.15.145:0/2505696270 shutdown_connections 2024-03-28T18:42:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.725+0000 7f04e9f98740 1 -- 172.21.15.145:0/2505696270 wait complete. 2024-03-28T18:42:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.725+0000 7f04e9f98740 1 -- start start 2024-03-28T18:42:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.725+0000 7f04e9f98740 1 Processor -- start 2024-03-28T18:42:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d821819b20 0x55d821823eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d8218243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.727 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d821822630 con 0x55d82181f680 2024-03-28T18:42:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d82173ad50 con 0x55d82181a510 2024-03-28T18:42:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8218227b0 con 0x55d821819b20 2024-03-28T18:42:04.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.726+0000 7f04e9f98740 10 client.0 _ll_register_callbacks cb 0x55d82173a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:04.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.727+0000 7f04e5950640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.727+0000 7f04e5950640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34264/0 (socket says 172.21.15.145:34264) 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 -- 172.21.15.145:0/3935277127 learned_addr learned my addr 172.21.15.145:0/3935277127 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 -- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d821819b20 msgr2=0x55d821823eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d821819b20 0x55d821823eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 -- 172.21.15.145:0/3935277127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 msgr2=0x55d8218243f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d8218243f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.728+0000 7f04e5950640 1 -- 172.21.15.145:0/3935277127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d82155f620 con 0x55d82181f680 2024-03-28T18:42:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.730+0000 7f04e494e640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 0x7f04cc0a4ec0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d821825390 tx=0x7f04dc006bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.732+0000 7f04e5950640 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04d80c6230 tx=0x7f04d80bdbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d06be20573df652 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.733+0000 7f04c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:42:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.735+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d80c2110 con 0x55d82181f680 2024-03-28T18:42:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.735+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04d0005500 con 0x55d82181f680 2024-03-28T18:42:04.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.735+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f04d8034bc0 con 0x55d82181f680 2024-03-28T18:42:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.739+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04d80c6760 con 0x55d82181f680 2024-03-28T18:42:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.739+0000 7f04e9f98740 10 client.15756 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:04.740 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.739+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d821822630 con 0x55d82181f680 2024-03-28T18:42:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.743+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d0005500 con 0x55d82181f680 2024-03-28T18:42:04.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f04d80c0070 con 0x55d82181f680 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 10 client.15756 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 20 client.15756 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82182fc60 0x55d8218320a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 4 client.15756 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:42:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d82178c050 con 0x55d82182fc60 2024-03-28T18:42:04.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.744+0000 7f04e514f640 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82182fc60 0x55d8218320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.745+0000 7f04e514f640 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82182fc60 0x55d8218320a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=426739ea047ac65a server_cookie=227ec699d8469b8d in_seq=0 out_seq=0 2024-03-28T18:42:04.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.746+0000 7f04c8ff9640 10 client.15756 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.748+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3935277127 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d82178c050 con 0x55d82182fc60 2024-03-28T18:42:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.748+0000 7f04c8ff9640 10 client.15756 handle_command_reply: tid=0 2024-03-28T18:42:04.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.749+0000 7f04e9f98740 1 client.15756 shutdown 2024-03-28T18:42:04.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.752+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 msgr2=0x55d82182b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.752+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04d80c6230 tx=0x7f04d80bdbc0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 shutdown_connections 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82182fc60 0x55d8218320a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181f680 0x55d82182b430 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d8218243f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3935277127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d821819b20 0x55d821823eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.758+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 >> 172.21.15.145:0/3935277127 conn(0x55d821815090 msgr2=0x55d821814290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.759+0000 7f04c8ff9640 0 client.15756 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:04.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.762+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 shutdown_connections 2024-03-28T18:42:04.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.762+0000 7f04e9f98740 1 -- 172.21.15.145:0/3935277127 wait complete. 2024-03-28T18:42:04.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.763+0000 7f04e9f98740 20 client.15756 trim_cache size 0 max 16384 2024-03-28T18:42:04.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.796+0000 7f04e9f98740 1 Processor -- start 2024-03-28T18:42:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 -- start start 2024-03-28T18:42:04.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181a510 0x55d82182b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d8218369a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01b0640 con 0x55d82181a510 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01b1fc0 con 0x55d821819b20 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.797+0000 7f04e9f98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01b0ca0 con 0x55d82181f680 2024-03-28T18:42:04.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42710/0 (socket says 172.21.15.145:42710) 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 -- 172.21.15.145:0/2447057320 learned_addr learned my addr 172.21.15.145:0/2447057320 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 -- 172.21.15.145:0/2447057320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 msgr2=0x55d8218369a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d8218369a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 -- 172.21.15.145:0/2447057320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181a510 msgr2=0x55d82182b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181a510 0x55d82182b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.798+0000 7f04e514f640 1 -- 172.21.15.145:0/2447057320 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d821815e00 con 0x55d821819b20 2024-03-28T18:42:04.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.799+0000 7f04e514f640 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04d0000c00 tx=0x7f04d0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20a5b5be3c19279 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.799+0000 7f04caffd640 1 -- 172.21.15.145:0/2447057320 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0039070 con 0x55d821819b20 2024-03-28T18:42:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.799+0000 7f04caffd640 1 -- 172.21.15.145:0/2447057320 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d001ee80 con 0x55d821819b20 2024-03-28T18:42:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.800+0000 7f04e9f98740 1 -- 172.21.15.145:0/2447057320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 msgr2=0x55d821818a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.800+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04d0000c00 tx=0x7f04d0005f10 comp rx=0 tx=0).stop 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 -- 172.21.15.145:0/2447057320 shutdown_connections 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d8218369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d82181a510 0x55d82182b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 --2- 172.21.15.145:0/2447057320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d821819b20 0x55d821818a00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 -- 172.21.15.145:0/2447057320 >> 172.21.15.145:0/2447057320 conn(0x55d821815090 msgr2=0x55d8218182f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 -- 172.21.15.145:0/2447057320 shutdown_connections 2024-03-28T18:42:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 -- 172.21.15.145:0/2447057320 wait complete. 2024-03-28T18:42:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.802+0000 7f04e9f98740 1 -- start start 2024-03-28T18:42:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 Processor -- start 2024-03-28T18:42:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d8218292f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d821829830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01b1960 con 0x55d821819b20 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01b1300 con 0x55d82181a510 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e01ae800 con 0x55d82181f680 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.803+0000 7f04e9f98740 10 client.0 _ll_register_callbacks cb 0x7f04d80ba350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.804+0000 7f04e5950640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.804+0000 7f04e5950640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:58712/0 (socket says 172.21.15.145:58712) 2024-03-28T18:42:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.804+0000 7f04e5950640 1 -- 172.21.15.145:0/3047445239 learned_addr learned my addr 172.21.15.145:0/3047445239 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:04.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.806+0000 7f04e5950640 1 -- 172.21.15.145:0/3047445239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 msgr2=0x55d821829830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:42:04.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.806+0000 7f04e5950640 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d821829830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.806+0000 7f04e5950640 1 -- 172.21.15.145:0/3047445239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 msgr2=0x55d8218292f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:04.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.806+0000 7f04e5950640 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d8218292f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.806+0000 7f04e5950640 1 -- 172.21.15.145:0/3047445239 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d82182ae30 con 0x55d82181f680 2024-03-28T18:42:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.807+0000 7f04e5950640 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04d80bdf50 tx=0x7f04d8034df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24dab4ce91beda0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.808+0000 7f04c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:42:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.809+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d80c2110 con 0x55d82181f680 2024-03-28T18:42:04.810 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.810+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f04d80bd410 con 0x55d82181f680 2024-03-28T18:42:04.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.811+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8217863a0 con 0x55d82181f680 2024-03-28T18:42:04.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.813+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04d80d2100 con 0x55d82181f680 2024-03-28T18:42:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.814+0000 7f04e9f98740 10 client.25606 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:04.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.814+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d82188bd40 con 0x55d82181f680 2024-03-28T18:42:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d8217863a0 con 0x55d82181f680 2024-03-28T18:42:04.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f04d80cd020 con 0x55d82181f680 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04e9f98740 10 client.25606 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04e9f98740 10 client.25606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04e9f98740 20 client.25606 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.816+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82188c3e0 0x55d82188e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.817+0000 7f04e9f98740 4 client.25606 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:42:04.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.817+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d821704710 con 0x55d82188c3e0 2024-03-28T18:42:04.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.821+0000 7f04e514f640 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82188c3e0 0x55d82188e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.826+0000 7f04e514f640 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82188c3e0 0x55d82188e860 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c6002b335253fa server_cookie=ad683bb9f6b430d4 in_seq=0 out_seq=0 2024-03-28T18:42:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:04.826+0000 7f04c8ff9640 10 client.25606 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:05.033 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:04 smithi145 ceph-mon[32151]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 596 MiB/s wr, 1.02k op/s 2024-03-28T18:42:05.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:04 smithi192 ceph-mon[36589]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 596 MiB/s wr, 1.02k op/s 2024-03-28T18:42:05.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:04 smithi164 ceph-mon[36248]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 596 MiB/s wr, 1.02k op/s 2024-03-28T18:42:07.035 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:06 smithi145 ceph-mon[32151]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 837 MiB/s wr, 1.25k op/s 2024-03-28T18:42:07.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:06 smithi192 ceph-mon[36589]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 837 MiB/s wr, 1.25k op/s 2024-03-28T18:42:07.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:06 smithi164 ceph-mon[36248]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 837 MiB/s wr, 1.25k op/s 2024-03-28T18:42:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04c8ff9640 1 -- 172.21.15.145:0/3047445239 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d821704710 con 0x55d82188c3e0 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04c8ff9640 10 client.25606 handle_command_reply: tid=0 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04e9f98740 1 client.25606 shutdown 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 msgr2=0x55d82182a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f04d80bdf50 tx=0x7f04d8034df0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.642+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 shutdown_connections 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d82188c3e0 0x55d82188e860 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d82181f680 0x55d82182a840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d82181a510 0x55d821829830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 --2- 172.21.15.145:0/3047445239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d821819b20 0x55d8218292f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 >> 172.21.15.145:0/3047445239 conn(0x55d821815090 msgr2=0x55d821818b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04c8ff9640 0 client.25606 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 shutdown_connections 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 1 -- 172.21.15.145:0/3047445239 wait complete. 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.643+0000 7f04e9f98740 20 client.25606 trim_cache size 0 max 16384 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:42:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:42:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:42:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.646+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 msgr2=0x7f04cc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.646+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 0x7f04cc0a4ec0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d821825390 tx=0x7f04dc006bb0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.646+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 msgr2=0x7f04e01a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.646+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04d8006010 tx=0x7f04d8003ba0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 shutdown_connections 2024-03-28T18:42:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04e0109390 0x7f04e01a53c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e00732b0 0x7f04e019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f04cc0a2a00 0x7f04cc0a4ec0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 --2- 172.21.15.145:0/2517481155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04e0072950 0x7f04e019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.648+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 >> 172.21.15.145:0/2517481155 conn(0x7f04e006e510 msgr2=0x7f04e010c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.649+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 shutdown_connections 2024-03-28T18:42:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:08.649+0000 7f04e6151640 1 -- 172.21.15.145:0/2517481155 wait complete. 2024-03-28T18:42:09.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:08 smithi145 ceph-mon[32151]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 1.14k op/s 2024-03-28T18:42:09.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:08 smithi192 ceph-mon[36589]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 1.14k op/s 2024-03-28T18:42:09.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:08 smithi164 ceph-mon[36248]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 678 MiB/s wr, 1.14k op/s 2024-03-28T18:42:11.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:10 smithi145 ceph-mon[32151]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 723 MiB/s wr, 1.32k op/s 2024-03-28T18:42:11.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:10 smithi192 ceph-mon[36589]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 723 MiB/s wr, 1.32k op/s 2024-03-28T18:42:11.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:10 smithi164 ceph-mon[36248]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 723 MiB/s wr, 1.32k op/s 2024-03-28T18:42:13.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:12 smithi164 ceph-mon[36248]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 709 MiB/s wr, 1.28k op/s 2024-03-28T18:42:13.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:12 smithi145 ceph-mon[32151]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 709 MiB/s wr, 1.28k op/s 2024-03-28T18:42:13.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:12 smithi192 ceph-mon[36589]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 709 MiB/s wr, 1.28k op/s 2024-03-28T18:42:15.041 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:14 smithi164 ceph-mon[36248]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 664 MiB/s wr, 1.14k op/s 2024-03-28T18:42:15.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:14 smithi192 ceph-mon[36589]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 664 MiB/s wr, 1.14k op/s 2024-03-28T18:42:15.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:14 smithi145 ceph-mon[32151]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 664 MiB/s wr, 1.14k op/s 2024-03-28T18:42:17.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:16 smithi192 ceph-mon[36589]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 830 MiB/s wr, 1.34k op/s 2024-03-28T18:42:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:16 smithi164 ceph-mon[36248]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 830 MiB/s wr, 1.34k op/s 2024-03-28T18:42:17.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:16 smithi145 ceph-mon[32151]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 830 MiB/s wr, 1.34k op/s 2024-03-28T18:42:19.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:18 smithi192 ceph-mon[36589]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:42:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:18 smithi164 ceph-mon[36248]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:42:19.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:18 smithi145 ceph-mon[32151]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:42:21.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:20 smithi192 ceph-mon[36589]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 MiB/s wr, 1.53k op/s 2024-03-28T18:42:21.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:20 smithi164 ceph-mon[36248]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 MiB/s wr, 1.53k op/s 2024-03-28T18:42:21.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:20 smithi145 ceph-mon[32151]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 MiB/s wr, 1.53k op/s 2024-03-28T18:42:23.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:22 smithi192 ceph-mon[36589]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:42:23.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:22 smithi164 ceph-mon[36248]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:42:23.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:22 smithi145 ceph-mon[32151]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:42:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.785+0000 7f738937e640 1 -- 172.21.15.145:0/4154090006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 msgr2=0x7f7384100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.785+0000 7f738937e640 1 --2- 172.21.15.145:0/4154090006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384100b10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7378009860 tx=0x7f737802bb90 comp rx=0 tx=0).stop 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 -- 172.21.15.145:0/4154090006 shutdown_connections 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 --2- 172.21.15.145:0/4154090006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 --2- 172.21.15.145:0/4154090006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384100b10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 --2- 172.21.15.145:0/4154090006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73840ff580 0x7f73840ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 -- 172.21.15.145:0/4154090006 >> 172.21.15.145:0/4154090006 conn(0x7f73840fae80 msgr2=0x7f73840fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:23.786 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 -- 172.21.15.145:0/4154090006 shutdown_connections 2024-03-28T18:42:23.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.786+0000 7f738937e640 1 -- 172.21.15.145:0/4154090006 wait complete. 2024-03-28T18:42:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.793+0000 7f738937e640 1 Processor -- start 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 -- start start 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73840ff580 0x7f7384071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384072840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841139c0 con 0x7f73840ff580 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384113840 con 0x7f73841006b0 2024-03-28T18:42:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.794+0000 7f738937e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384113b40 con 0x7f7384101050 2024-03-28T18:42:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.797+0000 7f73827fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.797+0000 7f73827fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44478/0 (socket says 172.21.15.145:44478) 2024-03-28T18:42:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.797+0000 7f73827fc640 1 -- 172.21.15.145:0/2456681667 learned_addr learned my addr 172.21.15.145:0/2456681667 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.797+0000 7f73837fe640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384072840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 -- 172.21.15.145:0/2456681667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 msgr2=0x7f7384072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384072840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 -- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73840ff580 msgr2=0x7f7384071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73840ff580 0x7f7384071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 -- 172.21.15.145:0/2456681667 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738410d4d0 con 0x7f73841006b0 2024-03-28T18:42:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73837fe640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:42:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.798+0000 7f73827fc640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f737802c1a0 tx=0x7f73780042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.799+0000 7f736bfff640 1 -- 172.21.15.145:0/2456681667 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378041020 con 0x7f73841006b0 2024-03-28T18:42:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.799+0000 7f736bfff640 1 -- 172.21.15.145:0/2456681667 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73780323b0 con 0x7f73841006b0 2024-03-28T18:42:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.799+0000 7f736bfff640 1 -- 172.21.15.145:0/2456681667 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7378039040 con 0x7f73841006b0 2024-03-28T18:42:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.803+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738410d760 con 0x7f73841006b0 2024-03-28T18:42:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.808+0000 7f736bfff640 1 -- 172.21.15.145:0/2456681667 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7378044020 con 0x7f73841006b0 2024-03-28T18:42:23.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.812+0000 7f736bfff640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 0x7f736c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.816+0000 7f7382ffd640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 0x7f736c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:23.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.831+0000 7f7382ffd640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 0x7f736c0a4ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7374005ab0 tx=0x7f7374005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.841+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738410dc50 con 0x7f73841006b0 2024-03-28T18:42:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:23.845+0000 7f736bfff640 1 -- 172.21.15.145:0/2456681667 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f73780c0150 con 0x7f73841006b0 2024-03-28T18:42:24.177 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.177+0000 7f738bf6d740 1 Processor -- start 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.177+0000 7f738bf6d740 1 -- start start 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396020cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960211e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563960171b10 con 0x56396020c920 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396011a900 con 0x563960212480 2024-03-28T18:42:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f738bf6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396016dbb0 con 0x56396020d310 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f73837fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f73837fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44490/0 (socket says 172.21.15.145:44490) 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.178+0000 7f73837fe640 1 -- 172.21.15.145:0/3107084012 learned_addr learned my addr 172.21.15.145:0/3107084012 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 -- 172.21.15.145:0/3107084012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 msgr2=0x563960211e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960211e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 -- 172.21.15.145:0/3107084012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 msgr2=0x56396020cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396020cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 -- 172.21.15.145:0/3107084012 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56395fe86210 con 0x563960212480 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.179+0000 7f73837fe640 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f737c007f30 tx=0x7f737c00c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ae5415056bb4f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.180+0000 7f73697fa640 1 -- 172.21.15.145:0/3107084012 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c014070 con 0x563960212480 2024-03-28T18:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.180+0000 7f73697fa640 1 -- 172.21.15.145:0/3107084012 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f737c00fd30 con 0x563960212480 2024-03-28T18:42:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.180+0000 7f73697fa640 1 -- 172.21.15.145:0/3107084012 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c00edf0 con 0x563960212480 2024-03-28T18:42:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.181+0000 7f738bf6d740 1 -- 172.21.15.145:0/3107084012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 msgr2=0x563960214910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.181+0000 7f738bf6d740 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f737c007f30 tx=0x7f737c00c750 comp rx=0 tx=0).stop 2024-03-28T18:42:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.185+0000 7f738bf6d740 1 -- 172.21.15.145:0/3107084012 shutdown_connections 2024-03-28T18:42:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.185+0000 7f738bf6d740 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x563960214910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.185+0000 7f738bf6d740 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960211e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.185+0000 7f738bf6d740 1 --2- 172.21.15.145:0/3107084012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396020cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.185+0000 7f738bf6d740 1 -- 172.21.15.145:0/3107084012 >> 172.21.15.145:0/3107084012 conn(0x563960207e90 msgr2=0x563960208290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.186+0000 7f738bf6d740 1 -- 172.21.15.145:0/3107084012 shutdown_connections 2024-03-28T18:42:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.186+0000 7f738bf6d740 1 -- 172.21.15.145:0/3107084012 wait complete. 2024-03-28T18:42:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.266+0000 7f738bf6d740 1 -- start start 2024-03-28T18:42:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.288+0000 7f738bf6d740 1 Processor -- start 2024-03-28T18:42:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960216cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960217230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563960215430 con 0x563960212480 2024-03-28T18:42:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396012dd50 con 0x56396020c920 2024-03-28T18:42:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.289+0000 7f738bf6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639602155b0 con 0x56396020d310 2024-03-28T18:42:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:57048/0 (socket says 172.21.15.145:57048) 2024-03-28T18:42:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 -- 172.21.15.145:0/2997504880 learned_addr learned my addr 172.21.15.145:0/2997504880 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 -- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 msgr2=0x563960217230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960217230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 -- 172.21.15.145:0/2997504880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 msgr2=0x563960216cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960216cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.290+0000 7f73837fe640 1 -- 172.21.15.145:0/2997504880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56395ff52620 con 0x563960212480 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.291+0000 7f73837fe640 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f737c000f80 tx=0x7f737c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae316d54253e5626 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.291+0000 7f7366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:42:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.294+0000 7f7366ff5640 1 -- 172.21.15.145:0/2997504880 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c014040 con 0x563960212480 2024-03-28T18:42:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.294+0000 7f7366ff5640 1 -- 172.21.15.145:0/2997504880 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f737c00f850 con 0x563960212480 2024-03-28T18:42:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.312+0000 7f738bf6d740 10 client.0 _ll_register_callbacks cb 0x56396012d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.313+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f737c00fd30 con 0x563960212480 2024-03-28T18:42:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.314+0000 7f7366ff5640 1 -- 172.21.15.145:0/2997504880 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f737c00eea0 con 0x563960212480 2024-03-28T18:42:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.314+0000 7f738bf6d740 10 client.15762 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.314+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({fsmap=31}) v3 -- 0x563960221450 con 0x563960212480 2024-03-28T18:42:24.353 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.353+0000 7f7366ff5640 1 -- 172.21.15.145:0/2997504880 <== mon.0 v2:172.21.15.145:3300/0 4 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f737c02b450 con 0x563960212480 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.353+0000 7f738bf6d740 10 client.15762 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.353+0000 7f738bf6d740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.354+0000 7f738bf6d740 20 client.15762 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.354+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563960222e40 0x563960225300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.354+0000 7f738bf6d740 4 client.15762 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:42:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.354+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56396017f050 con 0x563960222e40 2024-03-28T18:42:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.356+0000 7f7382ffd640 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563960222e40 0x563960225300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.365+0000 7f7382ffd640 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563960222e40 0x563960225300 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18957767974ed549 server_cookie=3d574e0b682d340b in_seq=0 out_seq=0 2024-03-28T18:42:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.366+0000 7f7366ff5640 10 client.15762 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.371+0000 7f7366ff5640 1 -- 172.21.15.145:0/2997504880 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56396017f050 con 0x563960222e40 2024-03-28T18:42:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.371+0000 7f7366ff5640 10 client.15762 handle_command_reply: tid=0 2024-03-28T18:42:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.371+0000 7f738bf6d740 1 client.15762 shutdown 2024-03-28T18:42:24.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.376+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 msgr2=0x56396021e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.376+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f737c000f80 tx=0x7f737c007c70 comp rx=0 tx=0).stop 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 shutdown_connections 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563960222e40 0x563960225300 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563960212480 0x56396021e270 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x563960217230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 --2- 172.21.15.145:0/2997504880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960216cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 >> 172.21.15.145:0/2997504880 conn(0x563960207e90 msgr2=0x563960207020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.378+0000 7f7366ff5640 0 client.15762 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.380+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 shutdown_connections 2024-03-28T18:42:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.380+0000 7f738bf6d740 1 -- 172.21.15.145:0/2997504880 wait complete. 2024-03-28T18:42:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.381+0000 7f738bf6d740 20 client.15762 trim_cache size 0 max 16384 2024-03-28T18:42:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.418+0000 7f738bf6d740 1 Processor -- start 2024-03-28T18:42:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.419+0000 7f738bf6d740 1 -- start start 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.420+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.420+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020d310 0x56396021e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.420+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563960212480 0x563960206790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f738bf6d740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841c1880 con 0x56396020d310 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f738bf6d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841c3200 con 0x56396020c920 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f738bf6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841c1ee0 con 0x563960212480 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f7382ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f7382ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44502/0 (socket says 172.21.15.145:44502) 2024-03-28T18:42:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f7382ffd640 1 -- 172.21.15.145:0/118347621 learned_addr learned my addr 172.21.15.145:0/118347621 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.421+0000 7f73837fe640 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563960212480 0x563960206790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.423+0000 7f7382ffd640 1 -- 172.21.15.145:0/118347621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563960212480 msgr2=0x563960206790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.423+0000 7f7382ffd640 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563960212480 0x563960206790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.423+0000 7f7382ffd640 1 -- 172.21.15.145:0/118347621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020d310 msgr2=0x56396021e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.423+0000 7f7382ffd640 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020d310 0x56396021e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.423+0000 7f7382ffd640 1 -- 172.21.15.145:0/118347621 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563960208c00 con 0x56396020c920 2024-03-28T18:42:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.426+0000 7f7382ffd640 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x56396020db70 tx=0x7f7374011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae45dc601ff993b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.426+0000 7f7368ff9640 1 -- 172.21.15.145:0/118347621 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374024250 con 0x56396020c920 2024-03-28T18:42:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.426+0000 7f7368ff9640 1 -- 172.21.15.145:0/118347621 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f737402a070 con 0x56396020c920 2024-03-28T18:42:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.426+0000 7f7368ff9640 1 -- 172.21.15.145:0/118347621 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374024cb0 con 0x56396020c920 2024-03-28T18:42:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.430+0000 7f738bf6d740 1 -- 172.21.15.145:0/118347621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 msgr2=0x563960225230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.430+0000 7f738bf6d740 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x56396020db70 tx=0x7f7374011000 comp rx=0 tx=0).stop 2024-03-28T18:42:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.434+0000 7f738bf6d740 1 -- 172.21.15.145:0/118347621 shutdown_connections 2024-03-28T18:42:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.434+0000 7f738bf6d740 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563960212480 0x563960206790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.434+0000 7f738bf6d740 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020d310 0x56396021e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.434+0000 7f738bf6d740 1 --2- 172.21.15.145:0/118347621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56396020c920 0x563960225230 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.434+0000 7f738bf6d740 1 -- 172.21.15.145:0/118347621 >> 172.21.15.145:0/118347621 conn(0x563960207e90 msgr2=0x563960208270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.442+0000 7f738bf6d740 1 -- 172.21.15.145:0/118347621 shutdown_connections 2024-03-28T18:42:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.444+0000 7f738bf6d740 1 -- 172.21.15.145:0/118347621 wait complete. 2024-03-28T18:42:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.445+0000 7f738bf6d740 1 -- start start 2024-03-28T18:42:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.445+0000 7f738bf6d740 1 Processor -- start 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x56396021bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x56396021cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841c2ba0 con 0x56396020c920 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841c2540 con 0x563960212480 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73841bfa40 con 0x56396020d310 2024-03-28T18:42:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.447+0000 7f738bf6d740 10 client.0 _ll_register_callbacks cb 0x7f737c00e790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.450+0000 7f7382ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.450+0000 7f7382ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:57064/0 (socket says 172.21.15.145:57064) 2024-03-28T18:42:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.450+0000 7f7382ffd640 1 -- 172.21.15.145:0/1017588332 learned_addr learned my addr 172.21.15.145:0/1017588332 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.450+0000 7f73827fc640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x56396021bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.453+0000 7f73837fe640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x56396021cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.455+0000 7f7382ffd640 1 -- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 msgr2=0x56396021bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.455+0000 7f7382ffd640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x56396021bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.455+0000 7f7382ffd640 1 -- 172.21.15.145:0/1017588332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 msgr2=0x56396021cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.455+0000 7f7382ffd640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x56396021cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.455+0000 7f7382ffd640 1 -- 172.21.15.145:0/1017588332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56396021d1e0 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.456+0000 7f7382ffd640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7374009f90 tx=0x7f7374009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1691c50d1e309b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.457+0000 7f7366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.457+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374032070 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.457+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f737402a070 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.457+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f73740039c0 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.458+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f737403e050 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.459+0000 7f738bf6d740 10 client.15765 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.459+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56396021e4f0 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f737402a070 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f737402e880 con 0x56396020c920 2024-03-28T18:42:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 10 client.15765 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 20 client.15765 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56396027ee00 0x563960281220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 4 client.15765 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:42:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.460+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5639601793a0 con 0x56396027ee00 2024-03-28T18:42:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.465+0000 7f73827fc640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56396027ee00 0x563960281220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.475+0000 7f73827fc640 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56396027ee00 0x563960281220 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de702e44341de1f5 server_cookie=50ba8ec64cfe3249 in_seq=0 out_seq=0 2024-03-28T18:42:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:24.476+0000 7f7366ff5640 10 client.15765 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:25.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:24 smithi192 ceph-mon[36589]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.23k op/s 2024-03-28T18:42:25.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:24 smithi164 ceph-mon[36248]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.23k op/s 2024-03-28T18:42:25.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:24 smithi145 ceph-mon[32151]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.23k op/s 2024-03-28T18:42:27.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:26 smithi192 ceph-mon[36589]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 808 MiB/s wr, 1.36k op/s 2024-03-28T18:42:27.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:26 smithi164 ceph-mon[36248]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 808 MiB/s wr, 1.36k op/s 2024-03-28T18:42:27.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:26 smithi145 ceph-mon[32151]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 808 MiB/s wr, 1.36k op/s 2024-03-28T18:42:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.439+0000 7f7366ff5640 1 -- 172.21.15.145:0/1017588332 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5639601793a0 con 0x56396027ee00 2024-03-28T18:42:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.439+0000 7f7366ff5640 10 client.15765 handle_command_reply: tid=0 2024-03-28T18:42:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 client.15765 shutdown 2024-03-28T18:42:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 msgr2=0x56396021b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7374009f90 tx=0x7f7374009fc0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 shutdown_connections 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56396027ee00 0x563960281220 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563960212480 0x56396021cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56396020d310 0x56396021bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 --2- 172.21.15.145:0/1017588332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56396020c920 0x56396021b6a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 >> 172.21.15.145:0/1017588332 conn(0x563960207e90 msgr2=0x563960206790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f7366ff5640 0 client.15765 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 shutdown_connections 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.440+0000 7f738bf6d740 1 -- 172.21.15.145:0/1017588332 wait complete. 2024-03-28T18:42:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.441+0000 7f738bf6d740 20 client.15765 trim_cache size 0 max 16384 2024-03-28T18:42:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:42:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:42:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:42:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:42:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.442+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 msgr2=0x7f736c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.442+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 0x7f736c0a4ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7374005ab0 tx=0x7f7374005a20 comp rx=0 tx=0).stop 2024-03-28T18:42:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.442+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 msgr2=0x7f7384072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.442+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f737802c1a0 tx=0x7f73780042f0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 shutdown_connections 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f736c0a2a10 0x7f736c0a4ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7384101050 0x7f7384072840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73841006b0 0x7f7384072300 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 --2- 172.21.15.145:0/2456681667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73840ff580 0x7f7384071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.443+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 >> 172.21.15.145:0/2456681667 conn(0x7f73840fae80 msgr2=0x7f7384103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.444+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 shutdown_connections 2024-03-28T18:42:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:27.444+0000 7f738937e640 1 -- 172.21.15.145:0/2456681667 wait complete. 2024-03-28T18:42:29.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:28 smithi192 ceph-mon[36589]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.21k op/s 2024-03-28T18:42:29.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:28 smithi164 ceph-mon[36248]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.21k op/s 2024-03-28T18:42:29.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:28 smithi145 ceph-mon[32151]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 668 MiB/s wr, 1.21k op/s 2024-03-28T18:42:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:30 smithi192 ceph-mon[36589]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 725 MiB/s wr, 1.33k op/s 2024-03-28T18:42:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:30 smithi164 ceph-mon[36248]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 725 MiB/s wr, 1.33k op/s 2024-03-28T18:42:31.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:30 smithi145 ceph-mon[32151]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 725 MiB/s wr, 1.33k op/s 2024-03-28T18:42:33.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:32 smithi192 ceph-mon[36589]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 673 MiB/s wr, 1.07k op/s 2024-03-28T18:42:33.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:32 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:32 smithi164 ceph-mon[36248]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 673 MiB/s wr, 1.07k op/s 2024-03-28T18:42:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:32 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:33.284 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:32 smithi145 ceph-mon[32151]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 673 MiB/s wr, 1.07k op/s 2024-03-28T18:42:33.284 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:32 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:35.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:34 smithi192 ceph-mon[36589]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 639 MiB/s wr, 933 op/s 2024-03-28T18:42:35.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:34 smithi164 ceph-mon[36248]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 639 MiB/s wr, 933 op/s 2024-03-28T18:42:35.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:34 smithi145 ceph-mon[32151]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 639 MiB/s wr, 933 op/s 2024-03-28T18:42:36.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:36.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:36.158 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:35 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:42:36.184 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:36.184 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:36.184 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:35 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:42:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:36.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:35 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:42:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:36 smithi192 ceph-mon[36589]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 802 MiB/s wr, 1.17k op/s 2024-03-28T18:42:37.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:36 smithi164 ceph-mon[36248]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 802 MiB/s wr, 1.17k op/s 2024-03-28T18:42:37.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:36 smithi145 ceph-mon[32151]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 802 MiB/s wr, 1.17k op/s 2024-03-28T18:42:39.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:38 smithi192 ceph-mon[36589]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 627 MiB/s wr, 1.02k op/s 2024-03-28T18:42:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:38 smithi164 ceph-mon[36248]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 627 MiB/s wr, 1.02k op/s 2024-03-28T18:42:39.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:38 smithi145 ceph-mon[32151]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 627 MiB/s wr, 1.02k op/s 2024-03-28T18:42:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.764+0000 7f9563fff640 1 -- 172.21.15.145:0/662479327 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410da40 msgr2=0x7f9564114360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 --2- 172.21.15.145:0/662479327 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410da40 0x7f9564114360 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f955c009ca0 tx=0x7f955c02c140 comp rx=0 tx=0).stop 2024-03-28T18:42:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 -- 172.21.15.145:0/662479327 shutdown_connections 2024-03-28T18:42:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 --2- 172.21.15.145:0/662479327 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410da40 0x7f9564114360 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 --2- 172.21.15.145:0/662479327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410d080 0x7f956410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 --2- 172.21.15.145:0/662479327 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f956410bea0 0x7f956410c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.765+0000 7f9563fff640 1 -- 172.21.15.145:0/662479327 >> 172.21.15.145:0/662479327 conn(0x7f956406bcd0 msgr2=0x7f956406c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.775+0000 7f9563fff640 1 -- 172.21.15.145:0/662479327 shutdown_connections 2024-03-28T18:42:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.777+0000 7f9563fff640 1 -- 172.21.15.145:0/662479327 wait complete. 2024-03-28T18:42:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.778+0000 7f9563fff640 1 Processor -- start 2024-03-28T18:42:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.779+0000 7f9563fff640 1 -- start start 2024-03-28T18:42:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f956410bea0 0x7f95641a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 0x7f95641a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 0x7f95641a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564117ca0 con 0x7f956410d080 2024-03-28T18:42:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564117b20 con 0x7f956410da40 2024-03-28T18:42:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.780+0000 7f9563fff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564117e20 con 0x7f956410bea0 2024-03-28T18:42:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.782+0000 7f95637fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 0x7f95641a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.783+0000 7f95637fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 0x7f95641a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57922/0 (socket says 172.21.15.145:57922) 2024-03-28T18:42:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.783+0000 7f95637fe640 1 -- 172.21.15.145:0/4214413043 learned_addr learned my addr 172.21.15.145:0/4214413043 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.783+0000 7f95627fc640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 0x7f95641a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.785+0000 7f95627fc640 1 -- 172.21.15.145:0/4214413043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f956410bea0 msgr2=0x7f95641a23e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:42:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.785+0000 7f95627fc640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f956410bea0 0x7f95641a23e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.785+0000 7f95627fc640 1 -- 172.21.15.145:0/4214413043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 msgr2=0x7f95641a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.785+0000 7f95627fc640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 0x7f95641a9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.785+0000 7f95627fc640 1 -- 172.21.15.145:0/4214413043 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95641aa000 con 0x7f956410d080 2024-03-28T18:42:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.791+0000 7f95627fc640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 0x7f95641a2920 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9558009fd0 tx=0x7f955800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.797+0000 7f954bfff640 1 -- 172.21.15.145:0/4214413043 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558015020 con 0x7f956410d080 2024-03-28T18:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.797+0000 7f954bfff640 1 -- 172.21.15.145:0/4214413043 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9558011d10 con 0x7f956410d080 2024-03-28T18:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.797+0000 7f954bfff640 1 -- 172.21.15.145:0/4214413043 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955800f2e0 con 0x7f956410d080 2024-03-28T18:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.799+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95641aa2f0 con 0x7f956410d080 2024-03-28T18:42:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.799+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95641aa870 con 0x7f956410d080 2024-03-28T18:42:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.804+0000 7f956c03f740 1 Processor -- start 2024-03-28T18:42:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 -- start start 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e41750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e46820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 0x55f389e49320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389d9db10 con 0x55f389e41330 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389d46900 con 0x55f389e46e90 2024-03-28T18:42:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.808+0000 7f956c03f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389d99bb0 con 0x55f389e41d20 2024-03-28T18:42:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.812+0000 7f95637fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e41750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.812+0000 7f95637fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e41750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36580/0 (socket says 172.21.15.145:36580) 2024-03-28T18:42:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.812+0000 7f95637fe640 1 -- 172.21.15.145:0/2803851486 learned_addr learned my addr 172.21.15.145:0/2803851486 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.812+0000 7f954bfff640 1 -- 172.21.15.145:0/4214413043 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9558018020 con 0x7f956410d080 2024-03-28T18:42:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.814+0000 7f95627fc640 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 0x55f389e49320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f9562ffd640 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e46820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f95627fc640 1 -- 172.21.15.145:0/2803851486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 msgr2=0x55f389e46820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f95627fc640 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e46820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f95627fc640 1 -- 172.21.15.145:0/2803851486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 msgr2=0x55f389e41750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f95627fc640 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e41750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.817+0000 7f95627fc640 1 -- 172.21.15.145:0/2803851486 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f389ab2210 con 0x55f389e46e90 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.819+0000 7f954bfff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 0x7f954c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.819+0000 7f954bfff640 1 -- 172.21.15.145:0/4214413043 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f95580ce590 con 0x7f956410d080 2024-03-28T18:42:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.823+0000 7f95627fc640 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 0x55f389e49320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95580cbc50 tx=0x7f95580d4a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74a47989e1be7fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.826 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.824+0000 7f95637fe640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 0x7f954c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.828+0000 7f95497fa640 1 -- 172.21.15.145:0/2803851486 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558096dc0 con 0x55f389e46e90 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.829+0000 7f95497fa640 1 -- 172.21.15.145:0/2803851486 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f955809b220 con 0x55f389e46e90 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.830+0000 7f956c03f740 1 -- 172.21.15.145:0/2803851486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 msgr2=0x55f389e49320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.830+0000 7f956c03f740 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 0x55f389e49320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95580cbc50 tx=0x7f95580d4a90 comp rx=0 tx=0).stop 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.831+0000 7f956c03f740 1 -- 172.21.15.145:0/2803851486 shutdown_connections 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.831+0000 7f956c03f740 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e46e90 0x55f389e49320 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95580cbc50 tx=0x7f95580d4a90 comp rx=0 tx=0).stop 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.831+0000 7f956c03f740 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e46820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.831+0000 7f956c03f740 1 --2- 172.21.15.145:0/2803851486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e41750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.831+0000 7f956c03f740 1 -- 172.21.15.145:0/2803851486 >> 172.21.15.145:0/2803851486 conn(0x55f389e34090 msgr2=0x55f389e34490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:40.838 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.838+0000 7f95637fe640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 0x7f954c0a4ec0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f955c009a60 tx=0x7f955c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.842+0000 7f956c03f740 1 -- 172.21.15.145:0/2803851486 shutdown_connections 2024-03-28T18:42:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.846+0000 7f956c03f740 1 -- 172.21.15.145:0/2803851486 wait complete. 2024-03-28T18:42:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.846+0000 7f956c03f740 1 -- start start 2024-03-28T18:42:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.846+0000 7f956c03f740 1 Processor -- start 2024-03-28T18:42:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e3ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e401d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389e38aa0 con 0x55f389e41330 2024-03-28T18:42:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389d59d50 con 0x55f389e3fd50 2024-03-28T18:42:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f389e38c20 con 0x55f389e41d20 2024-03-28T18:42:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.847+0000 7f956c03f740 10 client.0 _ll_register_callbacks cb 0x55f389d59720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.849+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.850+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36586/0 (socket says 172.21.15.145:36586) 2024-03-28T18:42:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.850+0000 7f9562ffd640 1 -- 172.21.15.145:0/4101692652 learned_addr learned my addr 172.21.15.145:0/4101692652 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.850+0000 7f95637fe640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.855 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.854+0000 7f95627fc640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e3ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.855+0000 7f9562ffd640 1 -- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 msgr2=0x55f389e3ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.855+0000 7f9562ffd640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e3ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.855+0000 7f9562ffd640 1 -- 172.21.15.145:0/4101692652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 msgr2=0x55f389e401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.855+0000 7f9562ffd640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.855+0000 7f9562ffd640 1 -- 172.21.15.145:0/4101692652 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f389b7e620 con 0x55f389e41330 2024-03-28T18:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.858+0000 7f9562ffd640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f955400ae80 tx=0x7f955400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0c97d34156951e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.858+0000 7f9546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.860+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554015070 con 0x55f389e41330 2024-03-28T18:42:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.860+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f95540115f0 con 0x55f389e41330 2024-03-28T18:42:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.861+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f389dab050 con 0x55f389e41330 2024-03-28T18:42:40.865 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.865+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f955401f050 con 0x55f389e41330 2024-03-28T18:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.866+0000 7f956c03f740 10 client.15777 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.866+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f389e5efa0 con 0x55f389e41330 2024-03-28T18:42:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f389dab050 con 0x55f389e41330 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f955401aac0 con 0x55f389e41330 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 10 client.15777 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 10 client.15777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 20 client.15777 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389e60b30 0x55f389e62fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 4 client.15777 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.868+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f389da53a0 con 0x55f389e60b30 2024-03-28T18:42:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.869+0000 7f95627fc640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389e60b30 0x55f389e62fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.879+0000 7f95627fc640 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389e60b30 0x55f389e62fb0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7a9ea746a6beb server_cookie=c46500c074981316 in_seq=0 out_seq=0 2024-03-28T18:42:40.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.881+0000 7f9546ff5640 10 client.15777 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.888+0000 7f9546ff5640 1 -- 172.21.15.145:0/4101692652 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f389da53a0 con 0x55f389e60b30 2024-03-28T18:42:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.888+0000 7f9546ff5640 10 client.15777 handle_command_reply: tid=0 2024-03-28T18:42:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.890+0000 7f956c03f740 1 client.15777 shutdown 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.897+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 msgr2=0x55f389e3a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.897+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f955400ae80 tx=0x7f955400c520 comp rx=0 tx=0).stop 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 shutdown_connections 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389e60b30 0x55f389e62fb0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e3ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 --2- 172.21.15.145:0/4101692652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e41330 0x55f389e3a7e0 secure :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f955400ae80 tx=0x7f955400c520 comp rx=0 tx=0).stop 2024-03-28T18:42:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 >> 172.21.15.145:0/4101692652 conn(0x55f389e34090 msgr2=0x55f389e33290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.898+0000 7f9546ff5640 0 client.15777 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.901+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 shutdown_connections 2024-03-28T18:42:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.902+0000 7f956c03f740 1 -- 172.21.15.145:0/4101692652 wait complete. 2024-03-28T18:42:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.903+0000 7f956c03f740 20 client.15777 trim_cache size 0 max 16384 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.945+0000 7f956c03f740 1 Processor -- start 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.945+0000 7f956c03f740 1 -- start start 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.945+0000 7f956c03f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e62b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.945+0000 7f956c03f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e43440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.946+0000 7f956c03f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.946+0000 7f956c03f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955000fb80 con 0x55f389e63080 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.946+0000 7f956c03f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550011500 con 0x55f389e3fd50 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.946+0000 7f956c03f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95500101e0 con 0x55f389e41d20 2024-03-28T18:42:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.947+0000 7f95637fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.947+0000 7f95637fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:36600/0 (socket says 172.21.15.145:36600) 2024-03-28T18:42:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.947+0000 7f95637fe640 1 -- 172.21.15.145:0/3800563183 learned_addr learned my addr 172.21.15.145:0/3800563183 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.948+0000 7f95627fc640 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e43440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.949+0000 7f95637fe640 1 -- 172.21.15.145:0/3800563183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 msgr2=0x55f389e43440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.949+0000 7f95637fe640 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e43440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.949+0000 7f95637fe640 1 -- 172.21.15.145:0/3800563183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 msgr2=0x55f389e62b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:42:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.949+0000 7f95637fe640 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e62b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.949+0000 7f95637fe640 1 -- 172.21.15.145:0/3800563183 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f389e35000 con 0x55f389e63080 2024-03-28T18:42:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.953+0000 7f95637fe640 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f955c032630 tx=0x7f955c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead1e982cb2f0a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.954+0000 7f9548ff9640 1 -- 172.21.15.145:0/3800563183 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c039040 con 0x55f389e63080 2024-03-28T18:42:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.954+0000 7f9548ff9640 1 -- 172.21.15.145:0/3800563183 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f955c052070 con 0x55f389e63080 2024-03-28T18:42:40.954 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.954+0000 7f9548ff9640 1 -- 172.21.15.145:0/3800563183 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c04d850 con 0x55f389e63080 2024-03-28T18:42:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.956+0000 7f956c03f740 1 -- 172.21.15.145:0/3800563183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 msgr2=0x55f389e43980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.956+0000 7f956c03f740 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f955c032630 tx=0x7f955c03f000 comp rx=0 tx=0).stop 2024-03-28T18:42:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.959+0000 7f956c03f740 1 -- 172.21.15.145:0/3800563183 shutdown_connections 2024-03-28T18:42:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.959+0000 7f956c03f740 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e43980 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.959+0000 7f956c03f740 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.959+0000 7f956c03f740 1 --2- 172.21.15.145:0/3800563183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e62b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.959+0000 7f956c03f740 1 -- 172.21.15.145:0/3800563183 >> 172.21.15.145:0/3800563183 conn(0x55f389e34090 msgr2=0x55f389e34470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.967+0000 7f956c03f740 1 -- 172.21.15.145:0/3800563183 shutdown_connections 2024-03-28T18:42:40.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.970+0000 7f956c03f740 1 -- 172.21.15.145:0/3800563183 wait complete. 2024-03-28T18:42:40.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.971+0000 7f956c03f740 1 -- start start 2024-03-28T18:42:40.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.971+0000 7f956c03f740 1 Processor -- start 2024-03-28T18:42:40.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.971+0000 7f956c03f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e70fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e71e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550010ea0 con 0x55f389e63080 2024-03-28T18:42:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550010840 con 0x55f389e3fd50 2024-03-28T18:42:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955000dd40 con 0x55f389e41d20 2024-03-28T18:42:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.972+0000 7f956c03f740 10 client.0 _ll_register_callbacks cb 0x7f955400aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.976+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.976+0000 7f9562ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57942/0 (socket says 172.21.15.145:57942) 2024-03-28T18:42:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.976+0000 7f9562ffd640 1 -- 172.21.15.145:0/40821148 learned_addr learned my addr 172.21.15.145:0/40821148 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.976+0000 7f95627fc640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e70fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.976+0000 7f95637fe640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e71e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.977+0000 7f9562ffd640 1 -- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 msgr2=0x55f389e70fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.977+0000 7f9562ffd640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e70fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.977+0000 7f9562ffd640 1 -- 172.21.15.145:0/40821148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 msgr2=0x55f389e71e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.977+0000 7f9562ffd640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e71e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.978+0000 7f9562ffd640 1 -- 172.21.15.145:0/40821148 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f389e72400 con 0x55f389e3fd50 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.978+0000 7f95627fc640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e70fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:40.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.978+0000 7f95637fe640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e71e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.979+0000 7f9562ffd640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9554000ea0 tx=0x7f9554019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43c13add772805d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.980+0000 7f9546ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:42:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.980+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554028070 con 0x55f389e3fd50 2024-03-28T18:42:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.980+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f955401a850 con 0x55f389e3fd50 2024-03-28T18:42:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.981+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f955c052070 con 0x55f389e3fd50 2024-03-28T18:42:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.983+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9554031050 con 0x55f389e3fd50 2024-03-28T18:42:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.983+0000 7f956c03f740 10 client.25547 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.983+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f389ebb990 con 0x55f389e3fd50 2024-03-28T18:42:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f955c052070 con 0x55f389e3fd50 2024-03-28T18:42:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f9554003690 con 0x55f389e3fd50 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 10 client.25547 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 10 client.25547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 20 client.25547 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389ebc260 0x55f389ebe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 4 client.25547 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:42:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.985+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f389d23710 con 0x55f389ebc260 2024-03-28T18:42:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.988+0000 7f95627fc640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389ebc260 0x55f389ebe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.992+0000 7f9546ff5640 10 client.25547 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:40.992+0000 7f95627fc640 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389ebc260 0x55f389ebe680 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf6383f93f560d6 server_cookie=87b66b9613cb321c in_seq=0 out_seq=0 2024-03-28T18:42:41.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:40 smithi192 ceph-mon[36589]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T18:42:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:40 smithi164 ceph-mon[36248]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T18:42:41.279 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:40 smithi145 ceph-mon[32151]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.11k op/s 2024-03-28T18:42:43.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:42 smithi192 ceph-mon[36589]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 710 MiB/s wr, 1.02k op/s 2024-03-28T18:42:43.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:42 smithi164 ceph-mon[36248]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 710 MiB/s wr, 1.02k op/s 2024-03-28T18:42:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:42 smithi145 ceph-mon[32151]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 710 MiB/s wr, 1.02k op/s 2024-03-28T18:42:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.809+0000 7f9546ff5640 1 -- 172.21.15.145:0/40821148 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f389d23710 con 0x55f389ebc260 2024-03-28T18:42:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.809+0000 7f9546ff5640 10 client.25547 handle_command_reply: tid=0 2024-03-28T18:42:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.809+0000 7f956c03f740 1 client.25547 shutdown 2024-03-28T18:42:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.809+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 msgr2=0x55f389e70a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.809+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9554000ea0 tx=0x7f9554019680 comp rx=0 tx=0).stop 2024-03-28T18:42:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 shutdown_connections 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f389ebc260 0x55f389ebe680 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f389e63080 0x55f389e71e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f389e41d20 0x55f389e70fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 --2- 172.21.15.145:0/40821148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f389e3fd50 0x55f389e70a90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 >> 172.21.15.145:0/40821148 conn(0x55f389e34090 msgr2=0x55f389e43440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f9546ff5640 0 client.25547 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 shutdown_connections 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 1 -- 172.21.15.145:0/40821148 wait complete. 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.810+0000 7f956c03f740 20 client.25547 trim_cache size 0 max 16384 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:42:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:42:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.812+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 msgr2=0x7f954c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.812+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 0x7f954c0a4ec0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f955c009a60 tx=0x7f955c02e040 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.812+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 msgr2=0x7f95641a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.812+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 0x7f95641a2920 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9558009fd0 tx=0x7f955800b870 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.812+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 shutdown_connections 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.813+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956410da40 0x7f95641a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.813+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956410d080 0x7f95641a2920 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.813+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f954c0a2a00 0x7f954c0a4ec0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.813+0000 7f9563fff640 1 --2- 172.21.15.145:0/4214413043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f956410bea0 0x7f95641a23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.813+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 >> 172.21.15.145:0/4214413043 conn(0x7f956406bcd0 msgr2=0x7f9564110530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.814+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 shutdown_connections 2024-03-28T18:42:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:44.814+0000 7f9563fff640 1 -- 172.21.15.145:0/4214413043 wait complete. 2024-03-28T18:42:45.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:44 smithi192 ceph-mon[36589]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 1.06k op/s 2024-03-28T18:42:45.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:44 smithi164 ceph-mon[36248]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 1.06k op/s 2024-03-28T18:42:45.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:44 smithi145 ceph-mon[32151]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 711 MiB/s wr, 1.06k op/s 2024-03-28T18:42:47.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:46 smithi192 ceph-mon[36589]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 MiB/s wr, 1.52k op/s 2024-03-28T18:42:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:46 smithi164 ceph-mon[36248]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 MiB/s wr, 1.52k op/s 2024-03-28T18:42:47.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:46 smithi145 ceph-mon[32151]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 MiB/s wr, 1.52k op/s 2024-03-28T18:42:49.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:48 smithi192 ceph-mon[36589]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 643 MiB/s wr, 1.34k op/s 2024-03-28T18:42:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:48 smithi164 ceph-mon[36248]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 643 MiB/s wr, 1.34k op/s 2024-03-28T18:42:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:48 smithi145 ceph-mon[32151]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 643 MiB/s wr, 1.34k op/s 2024-03-28T18:42:51.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:50 smithi192 ceph-mon[36589]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 772 MiB/s wr, 1.50k op/s 2024-03-28T18:42:51.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:50 smithi164 ceph-mon[36248]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 772 MiB/s wr, 1.50k op/s 2024-03-28T18:42:51.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:50 smithi145 ceph-mon[32151]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 772 MiB/s wr, 1.50k op/s 2024-03-28T18:42:53.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:52 smithi192 ceph-mon[36589]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 711 MiB/s wr, 1.38k op/s 2024-03-28T18:42:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:52 smithi164 ceph-mon[36248]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 711 MiB/s wr, 1.38k op/s 2024-03-28T18:42:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:52 smithi145 ceph-mon[32151]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 711 MiB/s wr, 1.38k op/s 2024-03-28T18:42:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:54 smithi164 ceph-mon[36248]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 657 MiB/s wr, 1.31k op/s 2024-03-28T18:42:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:54 smithi145 ceph-mon[32151]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 657 MiB/s wr, 1.31k op/s 2024-03-28T18:42:55.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:54 smithi192 ceph-mon[36589]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 657 MiB/s wr, 1.31k op/s 2024-03-28T18:42:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:56 smithi164 ceph-mon[36248]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 802 MiB/s wr, 1.59k op/s 2024-03-28T18:42:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:56 smithi145 ceph-mon[32151]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 802 MiB/s wr, 1.59k op/s 2024-03-28T18:42:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:56 smithi192 ceph-mon[36589]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 802 MiB/s wr, 1.59k op/s 2024-03-28T18:42:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.475+0000 7eff952d5640 1 -- 172.21.15.145:0/2424096944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff900732b0 msgr2=0x7eff90108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.475+0000 7eff952d5640 1 --2- 172.21.15.145:0/2424096944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff900732b0 0x7eff90108cd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff8001a910 tx=0x7eff8003cca0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.484+0000 7eff952d5640 1 -- 172.21.15.145:0/2424096944 shutdown_connections 2024-03-28T18:42:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.484+0000 7eff952d5640 1 --2- 172.21.15.145:0/2424096944 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90109390 0x7eff9010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.484+0000 7eff952d5640 1 --2- 172.21.15.145:0/2424096944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff900732b0 0x7eff90108cd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.484+0000 7eff952d5640 1 --2- 172.21.15.145:0/2424096944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff90072950 0x7eff90072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.484+0000 7eff952d5640 1 -- 172.21.15.145:0/2424096944 >> 172.21.15.145:0/2424096944 conn(0x7eff9006e510 msgr2=0x7eff90070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.492+0000 7eff952d5640 1 -- 172.21.15.145:0/2424096944 shutdown_connections 2024-03-28T18:42:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.492+0000 7eff952d5640 1 -- 172.21.15.145:0/2424096944 wait complete. 2024-03-28T18:42:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.492+0000 7eff952d5640 1 Processor -- start 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 -- start start 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff900732b0 0x7eff90119720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90109390 0x7eff901c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90111810 con 0x7eff90109390 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90111690 con 0x7eff900732b0 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.496+0000 7eff952d5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90111990 con 0x7eff90072950 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.497+0000 7eff8ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.497+0000 7eff8ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50266/0 (socket says 172.21.15.145:50266) 2024-03-28T18:42:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.497+0000 7eff8ffff640 1 -- 172.21.15.145:0/292700848 learned_addr learned my addr 172.21.15.145:0/292700848 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.498+0000 7eff8ffff640 1 -- 172.21.15.145:0/292700848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff900732b0 msgr2=0x7eff90119720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.498+0000 7eff8ffff640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff900732b0 0x7eff90119720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.498+0000 7eff8ffff640 1 -- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90109390 msgr2=0x7eff901c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.498+0000 7eff8ffff640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90109390 0x7eff901c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.498+0000 7eff8ffff640 1 -- 172.21.15.145:0/292700848 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff901c7bb0 con 0x7eff90072950 2024-03-28T18:42:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.509+0000 7eff8ffff640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c8c0 tx=0x7eff8400cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.512+0000 7eff8d7fa640 1 -- 172.21.15.145:0/292700848 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004220 con 0x7eff90072950 2024-03-28T18:42:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.512+0000 7eff8d7fa640 1 -- 172.21.15.145:0/292700848 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff840043c0 con 0x7eff90072950 2024-03-28T18:42:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.512+0000 7eff8d7fa640 1 -- 172.21.15.145:0/292700848 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004a00 con 0x7eff90072950 2024-03-28T18:42:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.512+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff901c7ea0 con 0x7eff90072950 2024-03-28T18:42:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.512+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff901c82e0 con 0x7eff90072950 2024-03-28T18:42:58.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.519+0000 7eff991f8740 1 Processor -- start 2024-03-28T18:42:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 -- start start 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f797510 0x55ad9f79c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f79eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f6fbb10 con 0x55ad9f797510 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f6a4900 con 0x55ad9f79c680 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.520+0000 7eff991f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f6f7bb0 con 0x55ad9f796b20 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50276/0 (socket says 172.21.15.145:50276) 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 -- 172.21.15.145:0/1472327144 learned_addr learned my addr 172.21.15.145:0/1472327144 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f79eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 -- 172.21.15.145:0/1472327144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 msgr2=0x55ad9f79eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f79eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 -- 172.21.15.145:0/1472327144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f797510 msgr2=0x55ad9f79c010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:42:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f797510 0x55ad9f79c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.521+0000 7eff94ad4640 1 -- 172.21.15.145:0/1472327144 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad9f410210 con 0x55ad9f796b20 2024-03-28T18:42:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.522+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff88009d00 tx=0x7eff8800b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7d8e6f582e6236b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.527+0000 7eff8d7fa640 1 -- 172.21.15.145:0/292700848 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7eff84004ba0 con 0x7eff90072950 2024-03-28T18:42:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.528+0000 7eff8d7fa640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 0x7eff780a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.529+0000 7eff8d7fa640 1 -- 172.21.15.145:0/292700848 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7eff840cea80 con 0x7eff90072950 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.530+0000 7eff7e7fc640 1 -- 172.21.15.145:0/1472327144 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8800bb30 con 0x55ad9f796b20 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.530+0000 7eff7e7fc640 1 -- 172.21.15.145:0/1472327144 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff88017070 con 0x55ad9f796b20 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 -- 172.21.15.145:0/1472327144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 msgr2=0x55ad9f796f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff88009d00 tx=0x7eff8800b420 comp rx=0 tx=0).stop 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 -- 172.21.15.145:0/1472327144 shutdown_connections 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f79eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f797510 0x55ad9f79c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 --2- 172.21.15.145:0/1472327144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f796b20 0x55ad9f796f40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.531+0000 7eff991f8740 1 -- 172.21.15.145:0/1472327144 >> 172.21.15.145:0/1472327144 conn(0x55ad9f792090 msgr2=0x55ad9f792490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.536+0000 7eff991f8740 1 -- 172.21.15.145:0/1472327144 shutdown_connections 2024-03-28T18:42:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.536+0000 7eff8ffff640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 0x7eff780a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.536+0000 7eff991f8740 1 -- 172.21.15.145:0/1472327144 wait complete. 2024-03-28T18:42:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.537+0000 7eff991f8740 1 -- start start 2024-03-28T18:42:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.537+0000 7eff991f8740 1 Processor -- start 2024-03-28T18:42:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f79c680 0x55ad9f7a1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f79f680 con 0x55ad9f7a63a0 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f6b7d50 con 0x55ad9f797510 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f79f800 con 0x55ad9f79c680 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff991f8740 10 client.0 _ll_register_callbacks cb 0x55ad9f6b7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:44646/0 (socket says 172.21.15.145:44646) 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.538+0000 7eff94ad4640 1 -- 172.21.15.145:0/1470023973 learned_addr learned my addr 172.21.15.145:0/1470023973 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.539+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.540+0000 7eff94ad4640 1 -- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f79c680 msgr2=0x55ad9f7a1370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:42:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.540+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f79c680 0x55ad9f7a1370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.540+0000 7eff94ad4640 1 -- 172.21.15.145:0/1470023973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 msgr2=0x55ad9f7a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.540+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.540+0000 7eff94ad4640 1 -- 172.21.15.145:0/1470023973 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad9f4dc620 con 0x55ad9f7a63a0 2024-03-28T18:42:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.544+0000 7eff94ad4640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7eff88004ba0 tx=0x7eff88004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6cc0f0aa2b27769 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.550+0000 7eff8ffff640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 0x7eff780a4da0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c890 tx=0x7eff84005480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.550+0000 7eff737fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.550+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88011e90 con 0x55ad9f7a63a0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.550+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad9f709050 con 0x55ad9f7a63a0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.550+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7eff88012430 con 0x55ad9f7a63a0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.551+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8802e050 con 0x55ad9f7a63a0 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.551+0000 7eff991f8740 10 client.15786 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.551+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ad9f7abb90 con 0x55ad9f7a63a0 2024-03-28T18:42:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.554+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ad9f709050 con 0x55ad9f7a63a0 2024-03-28T18:42:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.554+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7eff88011070 con 0x55ad9f7a63a0 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 10 client.15786 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 20 client.15786 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f7acff0 0x55ad9f7af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 4 client.15786 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:42:58.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.556+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad9f7033a0 con 0x55ad9f7acff0 2024-03-28T18:42:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.559+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f7acff0 0x55ad9f7af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.564+0000 7eff737fe640 10 client.15786 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.564+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f7acff0 0x55ad9f7af3e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca22bacdb3a487bb server_cookie=f83c7dfb35f7edf7 in_seq=0 out_seq=0 2024-03-28T18:42:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.568+0000 7eff737fe640 1 -- 172.21.15.145:0/1470023973 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad9f7033a0 con 0x55ad9f7acff0 2024-03-28T18:42:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.568+0000 7eff737fe640 10 client.15786 handle_command_reply: tid=0 2024-03-28T18:42:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.569+0000 7eff991f8740 1 client.15786 shutdown 2024-03-28T18:42:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 msgr2=0x55ad9f7a8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7eff88004ba0 tx=0x7eff88004af0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 shutdown_connections 2024-03-28T18:42:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f7acff0 0x55ad9f7af3e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7a8790 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f79c680 0x55ad9f7a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 --2- 172.21.15.145:0/1470023973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 >> 172.21.15.145:0/1470023973 conn(0x55ad9f792090 msgr2=0x55ad9f790a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.570+0000 7eff737fe640 0 client.15786 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.573+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 shutdown_connections 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.573+0000 7eff991f8740 1 -- 172.21.15.145:0/1470023973 wait complete. 2024-03-28T18:42:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.574+0000 7eff991f8740 20 client.15786 trim_cache size 0 max 16384 2024-03-28T18:42:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 Processor -- start 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 -- start start 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f79c680 0x55ad9f795830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7400fb00 con 0x55ad9f79c680 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74011480 con 0x55ad9f797510 2024-03-28T18:42:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.617+0000 7eff991f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74010160 con 0x55ad9f7a63a0 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:45288/0 (socket says 172.21.15.145:45288) 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 -- 172.21.15.145:0/2098638038 learned_addr learned my addr 172.21.15.145:0/2098638038 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 -- 172.21.15.145:0/2098638038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f7a63a0 msgr2=0x55ad9f7986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 -- 172.21.15.145:0/2098638038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f79c680 msgr2=0x55ad9f795830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f79c680 0x55ad9f795830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.618+0000 7eff94ad4640 1 -- 172.21.15.145:0/2098638038 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad9f7a16e0 con 0x55ad9f797510 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.619+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff8800bf70 tx=0x7eff8801d410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25fd4d7a99debf2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.620+0000 7eff7dffb640 1 -- 172.21.15.145:0/2098638038 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8800b920 con 0x55ad9f797510 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.620+0000 7eff7dffb640 1 -- 172.21.15.145:0/2098638038 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff88017040 con 0x55ad9f797510 2024-03-28T18:42:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.620+0000 7eff7dffb640 1 -- 172.21.15.145:0/2098638038 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8801d6e0 con 0x55ad9f797510 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 -- 172.21.15.145:0/2098638038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 msgr2=0x55ad9f7952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff8800bf70 tx=0x7eff8801d410 comp rx=0 tx=0).stop 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 -- 172.21.15.145:0/2098638038 shutdown_connections 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f79c680 0x55ad9f795830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 --2- 172.21.15.145:0/2098638038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f797510 0x55ad9f7952f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.621+0000 7eff991f8740 1 -- 172.21.15.145:0/2098638038 >> 172.21.15.145:0/2098638038 conn(0x55ad9f792090 msgr2=0x55ad9f792470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.627+0000 7eff991f8740 1 -- 172.21.15.145:0/2098638038 shutdown_connections 2024-03-28T18:42:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.627+0000 7eff991f8740 1 -- 172.21.15.145:0/2098638038 wait complete. 2024-03-28T18:42:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.627+0000 7eff991f8740 1 -- start start 2024-03-28T18:42:58.629 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.628+0000 7eff991f8740 1 Processor -- start 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f7bd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7be320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff74010e20 con 0x55ad9f7a63a0 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740107c0 con 0x55ad9f79c680 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7400dcc0 con 0x55ad9f797510 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff991f8740 10 client.0 _ll_register_callbacks cb 0x7eff88009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff94ad4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50314/0 (socket says 172.21.15.145:50314) 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.629+0000 7eff94ad4640 1 -- 172.21.15.145:0/2877865295 learned_addr learned my addr 172.21.15.145:0/2877865295 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f7bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 -- 172.21.15.145:0/2877865295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 msgr2=0x55ad9f7bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f7bd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 -- 172.21.15.145:0/2877865295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 msgr2=0x55ad9f7be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 -- 172.21.15.145:0/2877865295 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad9f7981e0 con 0x55ad9f797510 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.630+0000 7eff94ad4640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7eff8801d860 tx=0x7eff88012e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d162539fd753951 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.632+0000 7eff737fe640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.632+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88038070 con 0x55ad9f797510 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.632+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7eff88012870 con 0x55ad9f797510 2024-03-28T18:42:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.632+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff88017040 con 0x55ad9f797510 2024-03-28T18:42:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.636+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8800b920 con 0x55ad9f797510 2024-03-28T18:42:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.636+0000 7eff991f8740 10 client.25633 fetch_fsmap learned FSMap version 31 2024-03-28T18:42:58.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.636+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ad9f807c90 con 0x55ad9f797510 2024-03-28T18:42:58.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.640+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff88017040 con 0x55ad9f797510 2024-03-28T18:42:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.641+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7eff88031040 con 0x55ad9f797510 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 10 client.25633 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 10 client.25633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 20 client.25633 populate_metadata read hostname 'smithi145' 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f8083b0 0x55ad9f80a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 4 client.25633 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:42:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.642+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ad9f681710 con 0x55ad9f8083b0 2024-03-28T18:42:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.643+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f8083b0 0x55ad9f80a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.645+0000 7eff8f7fe640 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f8083b0 0x55ad9f80a7d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422b6527d54fbe63 server_cookie=fe380f05abe840a3 in_seq=0 out_seq=0 2024-03-28T18:42:58.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:42:58.648+0000 7eff737fe640 10 client.25633 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:42:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:42:58 smithi164 ceph-mon[36248]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 622 MiB/s wr, 1.05k op/s 2024-03-28T18:42:59.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:42:58 smithi145 ceph-mon[32151]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 622 MiB/s wr, 1.05k op/s 2024-03-28T18:42:59.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:42:58 smithi192 ceph-mon[36589]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 622 MiB/s wr, 1.05k op/s 2024-03-28T18:43:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:00 smithi164 ceph-mon[36248]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 MiB/s wr, 1.22k op/s 2024-03-28T18:43:01.282 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:00 smithi145 ceph-mon[32151]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 MiB/s wr, 1.22k op/s 2024-03-28T18:43:01.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:00 smithi192 ceph-mon[36589]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 779 MiB/s wr, 1.22k op/s 2024-03-28T18:43:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.112+0000 7eff737fe640 1 -- 172.21.15.145:0/2877865295 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ad9f681710 con 0x55ad9f8083b0 2024-03-28T18:43:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.112+0000 7eff737fe640 10 client.25633 handle_command_reply: tid=0 2024-03-28T18:43:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.112+0000 7eff991f8740 1 client.25633 shutdown 2024-03-28T18:43:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 msgr2=0x55ad9f7bce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7eff8801d860 tx=0x7eff88012e10 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 shutdown_connections 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad9f8083b0 0x55ad9f80a7d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad9f7a63a0 0x55ad9f7be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad9f79c680 0x55ad9f7bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 --2- 172.21.15.145:0/2877865295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad9f797510 0x55ad9f7bce40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 >> 172.21.15.145:0/2877865295 conn(0x55ad9f792090 msgr2=0x55ad9f7af000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff737fe640 0 client.25633 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 shutdown_connections 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.113+0000 7eff991f8740 1 -- 172.21.15.145:0/2877865295 wait complete. 2024-03-28T18:43:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.114+0000 7eff991f8740 20 client.25633 trim_cache size 0 max 16384 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.114+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 msgr2=0x7eff780a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.114+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 0x7eff780a4da0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c890 tx=0x7eff84005480 comp rx=0 tx=0).stop 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.114+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 msgr2=0x7eff901191e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.114+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c8c0 tx=0x7eff8400cd80 comp rx=0 tx=0).stop 2024-03-28T18:43:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 shutdown_connections 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90109390 0x7eff901c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff900732b0 0x7eff90119720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff780a28e0 0x7eff780a4da0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 --2- 172.21.15.145:0/292700848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90072950 0x7eff901191e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 >> 172.21.15.145:0/292700848 conn(0x7eff9006e510 msgr2=0x7eff9010c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 shutdown_connections 2024-03-28T18:43:02.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:02.115+0000 7eff952d5640 1 -- 172.21.15.145:0/292700848 wait complete. 2024-03-28T18:43:03.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:02 smithi164 ceph-mon[36248]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 723 MiB/s wr, 1.17k op/s 2024-03-28T18:43:03.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:02 smithi145 ceph-mon[32151]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 723 MiB/s wr, 1.17k op/s 2024-03-28T18:43:03.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:02 smithi192 ceph-mon[36589]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 723 MiB/s wr, 1.17k op/s 2024-03-28T18:43:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:04 smithi145 ceph-mon[32151]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 646 MiB/s wr, 1.10k op/s 2024-03-28T18:43:05.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:04 smithi192 ceph-mon[36589]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 646 MiB/s wr, 1.10k op/s 2024-03-28T18:43:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:04 smithi164 ceph-mon[36248]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 646 MiB/s wr, 1.10k op/s 2024-03-28T18:43:07.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:06 smithi145 ceph-mon[32151]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 850 MiB/s wr, 1.57k op/s 2024-03-28T18:43:07.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:06 smithi192 ceph-mon[36589]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 850 MiB/s wr, 1.57k op/s 2024-03-28T18:43:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:06 smithi164 ceph-mon[36248]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 850 MiB/s wr, 1.57k op/s 2024-03-28T18:43:09.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:08 smithi145 ceph-mon[32151]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 MiB/s wr, 1.16k op/s 2024-03-28T18:43:09.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:08 smithi192 ceph-mon[36589]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 MiB/s wr, 1.16k op/s 2024-03-28T18:43:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:08 smithi164 ceph-mon[36248]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 MiB/s wr, 1.16k op/s 2024-03-28T18:43:11.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:10 smithi145 ceph-mon[32151]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 758 MiB/s wr, 1.52k op/s 2024-03-28T18:43:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:11 smithi192 ceph-mon[36589]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 758 MiB/s wr, 1.52k op/s 2024-03-28T18:43:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:10 smithi164 ceph-mon[36248]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 758 MiB/s wr, 1.52k op/s 2024-03-28T18:43:13.264 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:13 smithi164 ceph-mon[36248]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 691 MiB/s wr, 1.42k op/s 2024-03-28T18:43:13.280 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:13 smithi145 ceph-mon[32151]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 691 MiB/s wr, 1.42k op/s 2024-03-28T18:43:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:13 smithi192 ceph-mon[36589]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 691 MiB/s wr, 1.42k op/s 2024-03-28T18:43:15.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:15 smithi145 ceph-mon[32151]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 609 MiB/s wr, 1.29k op/s 2024-03-28T18:43:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:15 smithi192 ceph-mon[36589]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 609 MiB/s wr, 1.29k op/s 2024-03-28T18:43:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:15 smithi164 ceph-mon[36248]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 609 MiB/s wr, 1.29k op/s 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 -- 172.21.15.145:0/871279374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da00fff70 msgr2=0x7f4da01023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 --2- 172.21.15.145:0/871279374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da00fff70 0x7f4da01023a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009ca0 tx=0x7f4d9802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 -- 172.21.15.145:0/871279374 shutdown_connections 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 --2- 172.21.15.145:0/871279374 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da0105290 0x7f4da0105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 --2- 172.21.15.145:0/871279374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da01028e0 0x7f4da0104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 --2- 172.21.15.145:0/871279374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da00fff70 0x7f4da01023a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.129+0000 7f4da63a9640 1 -- 172.21.15.145:0/871279374 >> 172.21.15.145:0/871279374 conn(0x7f4da00fbb70 msgr2=0x7f4da00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.130+0000 7f4da63a9640 1 -- 172.21.15.145:0/871279374 shutdown_connections 2024-03-28T18:43:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.130+0000 7f4da63a9640 1 -- 172.21.15.145:0/871279374 wait complete. 2024-03-28T18:43:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 Processor -- start 2024-03-28T18:43:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 -- start start 2024-03-28T18:43:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da00fff70 0x7f4da006ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da01028e0 0x7f4da0070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0113970 con 0x7f4da00fff70 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da01137f0 con 0x7f4da01028e0 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.131+0000 7f4da63a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0113af0 con 0x7f4da0105290 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56538/0 (socket says 172.21.15.145:56538) 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 -- 172.21.15.145:0/3392003945 learned_addr learned my addr 172.21.15.145:0/3392003945 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da01028e0 0x7f4da0070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da00fff70 0x7f4da006ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 -- 172.21.15.145:0/3392003945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da01028e0 msgr2=0x7f4da0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da01028e0 0x7f4da0070500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 -- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da00fff70 msgr2=0x7f4da006ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da00fff70 0x7f4da006ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.132+0000 7f4da4946640 1 -- 172.21.15.145:0/3392003945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0073680 con 0x7f4da0105290 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.133+0000 7f4da4946640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000d750 tx=0x7f4d9000dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.133+0000 7f4d9d7fa640 1 -- 172.21.15.145:0/3392003945 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90017070 con 0x7f4da0105290 2024-03-28T18:43:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.133+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da01ae680 con 0x7f4da0105290 2024-03-28T18:43:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.134+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da01aebd0 con 0x7f4da0105290 2024-03-28T18:43:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.134+0000 7f4d9d7fa640 1 -- 172.21.15.145:0/3392003945 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d90005500 con 0x7f4da0105290 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.134+0000 7f4d9d7fa640 1 -- 172.21.15.145:0/3392003945 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90004d90 con 0x7f4da0105290 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.140+0000 7f4d9d7fa640 1 -- 172.21.15.145:0/3392003945 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4d90010460 con 0x7f4da0105290 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.141+0000 7f4d9d7fa640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 0x7f4d880a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.141+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 0x7f4d880a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.141+0000 7f4d9d7fa640 1 -- 172.21.15.145:0/3392003945 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f4d900cf780 con 0x7f4da0105290 2024-03-28T18:43:16.142 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.142+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 0x7f4d880a4f50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4d9802c700 tx=0x7f4d98009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 Processor -- start 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 -- start start 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b73d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b7b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362ad8b10 con 0x55d362b73920 2024-03-28T18:43:16.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362a81900 con 0x55d362b74310 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da8f90740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362ad4bb0 con 0x55d362b79480 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da4946640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b7b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4d9f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da4946640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b7b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56548/0 (socket says 172.21.15.145:56548) 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4d9f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42566/0 (socket says 172.21.15.145:42566) 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.143+0000 7f4da4946640 1 -- 172.21.15.145:0/1382729250 learned_addr learned my addr 172.21.15.145:0/1382729250 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/1382729250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 msgr2=0x55d362b7b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b7b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/1382729250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 msgr2=0x55d362b73d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b73d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/1382729250 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3627ed210 con 0x55d362b74310 2024-03-28T18:43:16.145 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.144+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009bd0 tx=0x7f4d94011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb95d0666329cf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.149+0000 7f4d867fc640 1 -- 172.21.15.145:0/1382729250 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94004230 con 0x55d362b74310 2024-03-28T18:43:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.150+0000 7f4d867fc640 1 -- 172.21.15.145:0/1382729250 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d94009e40 con 0x55d362b74310 2024-03-28T18:43:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4d867fc640 1 -- 172.21.15.145:0/1382729250 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94004db0 con 0x55d362b74310 2024-03-28T18:43:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- 172.21.15.145:0/1382729250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 msgr2=0x55d362b78e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009bd0 tx=0x7f4d94011810 comp rx=0 tx=0).stop 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- 172.21.15.145:0/1382729250 shutdown_connections 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b7b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b74310 0x55d362b78e10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1382729250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b73d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- 172.21.15.145:0/1382729250 >> 172.21.15.145:0/1382729250 conn(0x55d362b6ee90 msgr2=0x55d362b6f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- 172.21.15.145:0/1382729250 shutdown_connections 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- 172.21.15.145:0/1382729250 wait complete. 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.151+0000 7f4da8f90740 1 -- start start 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 Processor -- start 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b7dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b74310 0x55d362b7e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362b7c480 con 0x55d362b73920 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362a94d50 con 0x55d362b79480 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d362b7c600 con 0x55d362b74310 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.152+0000 7f4da8f90740 10 client.0 _ll_register_callbacks cb 0x55d362a94720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4d9ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b7dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4d9ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b7dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58904/0 (socket says 172.21.15.145:58904) 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4d9ffff640 1 -- 172.21.15.145:0/1172453522 learned_addr learned my addr 172.21.15.145:0/1172453522 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b74310 0x55d362b7e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 -- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b74310 msgr2=0x55d362b7e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b74310 0x55d362b7e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 -- 172.21.15.145:0/1172453522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 msgr2=0x55d362b7dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b7dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.153+0000 7f4da4946640 1 -- 172.21.15.145:0/1172453522 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3628b9620 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.154+0000 7f4da4946640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b852f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009610 tx=0x7f4d9009c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d200a8070a2f8eec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.154+0000 7f4d83ff7640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.154+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90020040 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.154+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f4d900cf410 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.154+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d362ae6050 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.155+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d900a44b0 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.155+0000 7f4da8f90740 10 client.25568 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.155+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d362b996e0 con 0x55d362b79480 2024-03-28T18:43:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d362ae6050 con 0x55d362b79480 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4d900a4020 con 0x55d362b79480 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 10 client.25568 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 10 client.25568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 20 client.25568 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362b9adc0 0x55d362b9d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 4 client.25568 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.156+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d362ae03a0 con 0x55d362b9adc0 2024-03-28T18:43:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.157+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362b9adc0 0x55d362b9d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.158+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362b9adc0 0x55d362b9d1b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f059fb36859349ee server_cookie=21dce24151a14fd2 in_seq=0 out_seq=0 2024-03-28T18:43:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.158+0000 7f4d83ff7640 10 client.25568 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.161+0000 7f4d83ff7640 1 -- 172.21.15.145:0/1172453522 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d362ae03a0 con 0x55d362b9adc0 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.161+0000 7f4d83ff7640 10 client.25568 handle_command_reply: tid=0 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.161+0000 7f4da8f90740 1 client.25568 shutdown 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 msgr2=0x55d362b852f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b852f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009610 tx=0x7f4d9009c4d0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 shutdown_connections 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362b9adc0 0x55d362b9d1b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b852f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b74310 0x55d362b7e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 --2- 172.21.15.145:0/1172453522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b73920 0x55d362b7dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 >> 172.21.15.145:0/1172453522 conn(0x55d362b6ee90 msgr2=0x55d362b6e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.162+0000 7f4d83ff7640 0 client.25568 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.164+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 shutdown_connections 2024-03-28T18:43:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.164+0000 7f4da8f90740 1 -- 172.21.15.145:0/1172453522 wait complete. 2024-03-28T18:43:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.164+0000 7f4da8f90740 20 client.25568 trim_cache size 0 max 16384 2024-03-28T18:43:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.194+0000 7f4da8f90740 1 Processor -- start 2024-03-28T18:43:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.194+0000 7f4da8f90740 1 -- start start 2024-03-28T18:43:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.194+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.194+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b852b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b6dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4da8f90740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00fa10 con 0x55d362b74310 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4da8f90740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c011390 con 0x55d362b73920 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4da8f90740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c010070 con 0x55d362b79480 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4d9ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4d9ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42582/0 (socket says 172.21.15.145:42582) 2024-03-28T18:43:16.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.195+0000 7f4d9ffff640 1 -- 172.21.15.145:0/3111840349 learned_addr learned my addr 172.21.15.145:0/3111840349 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:16.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.196+0000 7f4d9ffff640 1 -- 172.21.15.145:0/3111840349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 msgr2=0x55d362b6dcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:43:16.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.196+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b6dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.196+0000 7f4d9ffff640 1 -- 172.21.15.145:0/3111840349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 msgr2=0x55d362b852b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.196+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b852b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.196+0000 7f4d9ffff640 1 -- 172.21.15.145:0/3111840349 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d362b9d830 con 0x55d362b73920 2024-03-28T18:43:16.204 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.204+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4d98037040 tx=0x7f4d98040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce45ecd7686c20de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.227+0000 7f4d85ffb640 1 -- 172.21.15.145:0/3111840349 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98039040 con 0x55d362b73920 2024-03-28T18:43:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.228+0000 7f4d85ffb640 1 -- 172.21.15.145:0/3111840349 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d98003e50 con 0x55d362b73920 2024-03-28T18:43:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.228+0000 7f4d85ffb640 1 -- 172.21.15.145:0/3111840349 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9803f470 con 0x55d362b73920 2024-03-28T18:43:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.234+0000 7f4da8f90740 1 -- 172.21.15.145:0/3111840349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 msgr2=0x55d362b6d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:16.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.234+0000 7f4da8f90740 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4d98037040 tx=0x7f4d98040000 comp rx=0 tx=0).stop 2024-03-28T18:43:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.238+0000 7f4da8f90740 1 -- 172.21.15.145:0/3111840349 shutdown_connections 2024-03-28T18:43:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.238+0000 7f4da8f90740 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b79480 0x55d362b6dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.238+0000 7f4da8f90740 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b852b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.238+0000 7f4da8f90740 1 --2- 172.21.15.145:0/3111840349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b73920 0x55d362b6d790 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.238+0000 7f4da8f90740 1 -- 172.21.15.145:0/3111840349 >> 172.21.15.145:0/3111840349 conn(0x55d362b6ee90 msgr2=0x55d362b75630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.240+0000 7f4da8f90740 1 -- 172.21.15.145:0/3111840349 shutdown_connections 2024-03-28T18:43:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.240+0000 7f4da8f90740 1 -- 172.21.15.145:0/3111840349 wait complete. 2024-03-28T18:43:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.241+0000 7f4da8f90740 1 -- start start 2024-03-28T18:43:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.242+0000 7f4da8f90740 1 Processor -- start 2024-03-28T18:43:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.243+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b73920 0x55d362b82f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.243+0000 7f4da8f90740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.243+0000 7f4d9f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.243+0000 7f4d9f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:58928/0 (socket says 172.21.15.145:58928) 2024-03-28T18:43:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.243+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/2456744733 learned_addr learned my addr 172.21.15.145:0/2456744733 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b73920 0x55d362b82f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c010d30 con 0x55d362b74310 2024-03-28T18:43:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0106d0 con 0x55d362b79480 2024-03-28T18:43:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c00dbd0 con 0x55d362b73920 2024-03-28T18:43:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.244+0000 7f4da8f90740 10 client.0 _ll_register_callbacks cb 0x7f4d900056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b73920 msgr2=0x55d362b82f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b73920 0x55d362b82f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/2456744733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 msgr2=0x55d362b844c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d362b84ab0 con 0x55d362b74310 2024-03-28T18:43:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.245+0000 7f4d9f7fe640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009bd0 tx=0x7f4d94003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219e279fa5feb863 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:16.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.257+0000 7f4d83ff7640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:43:16.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.257+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9401e040 con 0x55d362b74310 2024-03-28T18:43:16.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.257+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f4d94003b10 con 0x55d362b74310 2024-03-28T18:43:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.259+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d98003e50 con 0x55d362b74310 2024-03-28T18:43:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.262+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d94005640 con 0x55d362b74310 2024-03-28T18:43:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.262+0000 7f4da8f90740 10 client.15798 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.262+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d362bf6cd0 con 0x55d362b74310 2024-03-28T18:43:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.267+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d94028070 con 0x55d362b74310 2024-03-28T18:43:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.268+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f4d94018070 con 0x55d362b74310 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 10 client.15798 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 20 client.15798 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362bf73e0 0x55d362bf9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 4 client.15798 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.269+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f4d98003e50 con 0x55d362bf73e0 2024-03-28T18:43:16.271 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.270+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362bf73e0 0x55d362bf9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.282+0000 7f4d9ffff640 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362bf73e0 0x55d362bf9800 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aaa28eaa8212678 server_cookie=809492cdb213e756 in_seq=0 out_seq=0 2024-03-28T18:43:16.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:16.284+0000 7f4d83ff7640 10 client.15798 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:17 smithi145 ceph-mon[32151]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.77k op/s 2024-03-28T18:43:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:17 smithi192 ceph-mon[36589]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.77k op/s 2024-03-28T18:43:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:17 smithi164 ceph-mon[36248]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.77k op/s 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.197+0000 7f4d83ff7640 1 -- 172.21.15.145:0/2456744733 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f4d98003e50 con 0x55d362bf73e0 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.197+0000 7f4d83ff7640 10 client.15798 handle_command_reply: tid=0 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.197+0000 7f4da8f90740 1 client.15798 shutdown 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 msgr2=0x55d362b834b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009bd0 tx=0x7f4d94003820 comp rx=0 tx=0).stop 2024-03-28T18:43:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 shutdown_connections 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d362bf73e0 0x55d362bf9800 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d362b79480 0x55d362b844c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d362b74310 0x55d362b834b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 --2- 172.21.15.145:0/2456744733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d362b73920 0x55d362b82f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 >> 172.21.15.145:0/2456744733 conn(0x55d362b6ee90 msgr2=0x55d362b7dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4d83ff7640 0 client.15798 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 shutdown_connections 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 1 -- 172.21.15.145:0/2456744733 wait complete. 2024-03-28T18:43:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.198+0000 7f4da8f90740 20 client.15798 trim_cache size 0 max 16384 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.200+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 msgr2=0x7f4d880a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.200+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 0x7f4d880a4f50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4d9802c700 tx=0x7f4d98009920 comp rx=0 tx=0).stop 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.200+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 msgr2=0x7f4da01ae080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.200+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000d750 tx=0x7f4d9000dc10 comp rx=0 tx=0).stop 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 shutdown_connections 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f4d880a2a90 0x7f4d880a4f50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4da0105290 0x7f4da01ae080 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da01028e0 0x7f4da0070500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 --2- 172.21.15.145:0/3392003945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4da00fff70 0x7f4da006ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.201+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 >> 172.21.15.145:0/3392003945 conn(0x7f4da00fbb70 msgr2=0x7f4da01035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.202+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 shutdown_connections 2024-03-28T18:43:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:18.202+0000 7f4da63a9640 1 -- 172.21.15.145:0/3392003945 wait complete. 2024-03-28T18:43:19.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:19 smithi145 ceph-mon[32151]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 601 MiB/s wr, 1.34k op/s 2024-03-28T18:43:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:19 smithi192 ceph-mon[36589]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 601 MiB/s wr, 1.34k op/s 2024-03-28T18:43:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:19 smithi164 ceph-mon[36248]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 601 MiB/s wr, 1.34k op/s 2024-03-28T18:43:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:21 smithi192 ceph-mon[36589]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.62k op/s 2024-03-28T18:43:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:21 smithi164 ceph-mon[36248]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.62k op/s 2024-03-28T18:43:21.531 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:21 smithi145 ceph-mon[32151]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.62k op/s 2024-03-28T18:43:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:23 smithi192 ceph-mon[36589]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 668 MiB/s wr, 1.35k op/s 2024-03-28T18:43:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:23 smithi164 ceph-mon[36248]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 668 MiB/s wr, 1.35k op/s 2024-03-28T18:43:23.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:23 smithi145 ceph-mon[32151]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 668 MiB/s wr, 1.35k op/s 2024-03-28T18:43:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:25 smithi192 ceph-mon[36589]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 604 MiB/s wr, 1.26k op/s 2024-03-28T18:43:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:25 smithi164 ceph-mon[36248]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 604 MiB/s wr, 1.26k op/s 2024-03-28T18:43:25.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:25 smithi145 ceph-mon[32151]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 604 MiB/s wr, 1.26k op/s 2024-03-28T18:43:26.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:26 smithi192 ceph-mon[36589]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 827 MiB/s wr, 1.57k op/s 2024-03-28T18:43:26.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:26 smithi164 ceph-mon[36248]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 827 MiB/s wr, 1.57k op/s 2024-03-28T18:43:26.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:26 smithi145 ceph-mon[32151]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 827 MiB/s wr, 1.57k op/s 2024-03-28T18:43:29.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:28 smithi192 ceph-mon[36589]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:43:29.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:28 smithi164 ceph-mon[36248]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:43:29.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:28 smithi145 ceph-mon[32151]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:43:31.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:30 smithi192 ceph-mon[36589]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 726 MiB/s wr, 1.27k op/s 2024-03-28T18:43:31.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:30 smithi164 ceph-mon[36248]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 726 MiB/s wr, 1.27k op/s 2024-03-28T18:43:31.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:30 smithi145 ceph-mon[32151]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 726 MiB/s wr, 1.27k op/s 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.047+0000 7f05971df640 1 -- 172.21.15.145:0/1543202526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05901006b0 msgr2=0x7f0590100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.047+0000 7f05971df640 1 --2- 172.21.15.145:0/1543202526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05901006b0 0x7f0590100b10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f058001a910 tx=0x7f058003cc60 comp rx=0 tx=0).stop 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.048+0000 7f05971df640 1 -- 172.21.15.145:0/1543202526 shutdown_connections 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.048+0000 7f05971df640 1 --2- 172.21.15.145:0/1543202526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0590101050 0x7f0590105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.048+0000 7f05971df640 1 --2- 172.21.15.145:0/1543202526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05901006b0 0x7f0590100b10 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.048+0000 7f05971df640 1 --2- 172.21.15.145:0/1543202526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05900ff580 0x7f05900ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.048+0000 7f05971df640 1 -- 172.21.15.145:0/1543202526 >> 172.21.15.145:0/1543202526 conn(0x7f05900fae80 msgr2=0x7f05900fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.052+0000 7f05971df640 1 -- 172.21.15.145:0/1543202526 shutdown_connections 2024-03-28T18:43:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.052+0000 7f05971df640 1 -- 172.21.15.145:0/1543202526 wait complete. 2024-03-28T18:43:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 Processor -- start 2024-03-28T18:43:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 -- start start 2024-03-28T18:43:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05900ff580 0x7f05901956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05901006b0 0x7f0590195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05901092b0 con 0x7f05900ff580 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590109130 con 0x7f05901006b0 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f05971df640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590109430 con 0x7f0590101050 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.053+0000 7f059577c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:43060/0 (socket says 172.21.15.145:43060) 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 -- 172.21.15.145:0/675095680 learned_addr learned my addr 172.21.15.145:0/675095680 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:32.056 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f058ffff640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05901006b0 0x7f0590195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 -- 172.21.15.145:0/675095680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05901006b0 msgr2=0x7f0590195c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05901006b0 0x7f0590195c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 -- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05900ff580 msgr2=0x7f05901956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.054+0000 7f059577c640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05900ff580 0x7f05901956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.055+0000 7f059577c640 1 -- 172.21.15.145:0/675095680 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05901bf360 con 0x7f0590101050 2024-03-28T18:43:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.055+0000 7f059577c640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f058800cad0 tx=0x7f058800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.057+0000 7f058dffb640 1 -- 172.21.15.145:0/675095680 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058800ae20 con 0x7f0590101050 2024-03-28T18:43:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.057+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05901bf650 con 0x7f0590101050 2024-03-28T18:43:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.057+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05901bfb70 con 0x7f0590101050 2024-03-28T18:43:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.058+0000 7f058dffb640 1 -- 172.21.15.145:0/675095680 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0588005020 con 0x7f0590101050 2024-03-28T18:43:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.058+0000 7f058dffb640 1 -- 172.21.15.145:0/675095680 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058800f740 con 0x7f0590101050 2024-03-28T18:43:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f058dffb640 1 -- 172.21.15.145:0/675095680 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f058800f8e0 con 0x7f0590101050 2024-03-28T18:43:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 Processor -- start 2024-03-28T18:43:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 -- start start 2024-03-28T18:43:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e8075910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e7fd2b10 con 0x55e4e8073480 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.062+0000 7f0599dca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e7f7b900 con 0x55e4e806d920 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f0599dca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e7fcebb0 con 0x55e4e806e310 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44482/0 (socket says 172.21.15.145:44482) 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 -- 172.21.15.145:0/1173087669 learned_addr learned my addr 172.21.15.145:0/1173087669 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 -- 172.21.15.145:0/1173087669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 msgr2=0x55e4e8072e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 -- 172.21.15.145:0/1173087669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 msgr2=0x55e4e8075910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e8075910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.063+0000 7f058ffff640 1 -- 172.21.15.145:0/1173087669 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4e7ce7210 con 0x55e4e806d920 2024-03-28T18:43:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.064+0000 7f058ffff640 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0580004880 tx=0x7f05800053f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9679af48c3e16bff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.065+0000 7f0576ffd640 1 -- 172.21.15.145:0/1173087669 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580004030 con 0x55e4e806d920 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.065+0000 7f0576ffd640 1 -- 172.21.15.145:0/1173087669 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0580004f30 con 0x55e4e806d920 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.065+0000 7f0576ffd640 1 -- 172.21.15.145:0/1173087669 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058003dd70 con 0x55e4e806d920 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 -- 172.21.15.145:0/1173087669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 msgr2=0x55e4e806dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0580004880 tx=0x7f05800053f0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 -- 172.21.15.145:0/1173087669 shutdown_connections 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e8075910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 --2- 172.21.15.145:0/1173087669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e806dd40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.066+0000 7f0599dca740 1 -- 172.21.15.145:0/1173087669 >> 172.21.15.145:0/1173087669 conn(0x55e4e8068e90 msgr2=0x55e4e8069290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.070+0000 7f0599dca740 1 -- 172.21.15.145:0/1173087669 shutdown_connections 2024-03-28T18:43:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.070+0000 7f0599dca740 1 -- 172.21.15.145:0/1173087669 wait complete. 2024-03-28T18:43:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.070+0000 7f0599dca740 1 -- start start 2024-03-28T18:43:32.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.071+0000 7f0599dca740 1 Processor -- start 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.071+0000 7f0599dca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.071+0000 7f0599dca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e8078200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e8078200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e8078200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:51580/0 (socket says 172.21.15.145:51580) 2024-03-28T18:43:32.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:44494/0 (socket says 172.21.15.145:44494) 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0594f7b640 1 -- 172.21.15.145:0/2410859521 learned_addr learned my addr 172.21.15.145:0/2410859521 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8073480 0x55e4e807f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e80763d0 con 0x55e4e806e310 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e7f8ed50 con 0x55e4e806d920 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4e8076550 con 0x55e4e8073480 2024-03-28T18:43:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f0599dca740 10 client.0 _ll_register_callbacks cb 0x55e4e7f8e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8073480 0x55e4e807f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 -- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8073480 msgr2=0x55e4e807f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8073480 0x55e4e807f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 -- 172.21.15.145:0/2410859521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 msgr2=0x55e4e8078200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e8078200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.072+0000 7f058ffff640 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4e7db3620 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f058ffff640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f05800047e0 tx=0x7f058003d810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4915c601449f10da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f056ffff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800456d0 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f058004b040 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0580004f30 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.073+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0580045870 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.074+0000 7f0599dca740 10 client.25580 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.074+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e4e8082590 con 0x55e4e806d920 2024-03-28T18:43:32.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.074+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0580004f30 con 0x55e4e806d920 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f058006f050 con 0x55e4e806d920 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 10 client.25580 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 10 client.25580 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 20 client.25580 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80838e0 0x55e4e8085cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 4 client.25580 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.075+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4e7fe0050 con 0x55e4e80838e0 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.077+0000 7f058dffb640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 0x7f05780a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.077+0000 7f058dffb640 1 -- 172.21.15.145:0/675095680 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f05880cf620 con 0x7f0590101050 2024-03-28T18:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.080+0000 7f058ffff640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 0x7f05780a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.081+0000 7f0594f7b640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80838e0 0x55e4e8085cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.094+0000 7f0594f7b640 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80838e0 0x55e4e8085cd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b4a006a419afbee server_cookie=4523cf4fcf70e8ff in_seq=0 out_seq=0 2024-03-28T18:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.094+0000 7f056ffff640 10 client.25580 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.097+0000 7f058ffff640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 0x7f05780a4ec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0580004880 tx=0x7f0580005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.100+0000 7f056ffff640 1 -- 172.21.15.145:0/2410859521 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e4e7fe0050 con 0x55e4e80838e0 2024-03-28T18:43:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.100+0000 7f056ffff640 10 client.25580 handle_command_reply: tid=0 2024-03-28T18:43:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.100+0000 7f0599dca740 1 client.25580 shutdown 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 msgr2=0x55e4e8077cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f05800047e0 tx=0x7f058003d810 comp rx=0 tx=0).stop 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 shutdown_connections 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80838e0 0x55e4e8085cd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8073480 0x55e4e807f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e8078200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 --2- 172.21.15.145:0/2410859521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e806d920 0x55e4e8077cc0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f05800047e0 tx=0x7f058003d810 comp rx=0 tx=0).stop 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 >> 172.21.15.145:0/2410859521 conn(0x55e4e8068e90 msgr2=0x55e4e8068020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f056ffff640 0 client.25580 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 shutdown_connections 2024-03-28T18:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.105+0000 7f0599dca740 1 -- 172.21.15.145:0/2410859521 wait complete. 2024-03-28T18:43:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.106+0000 7f0599dca740 20 client.25580 trim_cache size 0 max 16384 2024-03-28T18:43:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.149+0000 7f0599dca740 1 Processor -- start 2024-03-28T18:43:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 -- start start 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e807f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8069270 0x55e4e808a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c00faf0 con 0x55e4e8073480 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c011470 con 0x55e4e8069270 2024-03-28T18:43:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.150+0000 7f0599dca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c010150 con 0x55e4e806e310 2024-03-28T18:43:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.151+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.151+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:43082/0 (socket says 172.21.15.145:43082) 2024-03-28T18:43:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.151+0000 7f0594f7b640 1 -- 172.21.15.145:0/1278594900 learned_addr learned my addr 172.21.15.145:0/1278594900 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.152+0000 7f0594f7b640 1 -- 172.21.15.145:0/1278594900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8069270 msgr2=0x55e4e808a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.152+0000 7f0594f7b640 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8069270 0x55e4e808a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.152+0000 7f0594f7b640 1 -- 172.21.15.145:0/1278594900 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 msgr2=0x55e4e807f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.152+0000 7f0594f7b640 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e807f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.152+0000 7f0594f7b640 1 -- 172.21.15.145:0/1278594900 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4e8069c00 con 0x55e4e806e310 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.157+0000 7f0594f7b640 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0584009a60 tx=0x7f0584005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52a717c9657a5820 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.158+0000 7f05767fc640 1 -- 172.21.15.145:0/1278594900 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584004250 con 0x55e4e806e310 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.158+0000 7f05767fc640 1 -- 172.21.15.145:0/1278594900 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0584018300 con 0x55e4e806e310 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.160+0000 7f0599dca740 1 -- 172.21.15.145:0/1278594900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 msgr2=0x55e4e8072cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.160+0000 7f0599dca740 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0584009a60 tx=0x7f0584005f30 comp rx=0 tx=0).stop 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 -- 172.21.15.145:0/1278594900 shutdown_connections 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8069270 0x55e4e808a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e8073480 0x55e4e807f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 --2- 172.21.15.145:0/1278594900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e806e310 0x55e4e8072cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 -- 172.21.15.145:0/1278594900 >> 172.21.15.145:0/1278594900 conn(0x55e4e8068e90 msgr2=0x55e4e8068020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 -- 172.21.15.145:0/1278594900 shutdown_connections 2024-03-28T18:43:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.161+0000 7f0599dca740 1 -- 172.21.15.145:0/1278594900 wait complete. 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.162+0000 7f0599dca740 1 -- start start 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.162+0000 7f0599dca740 1 Processor -- start 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 0x55e4e807ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e807d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 0x55e4e807e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c010e10 con 0x55e4e806e310 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c0107b0 con 0x55e4e8073480 2024-03-28T18:43:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0599dca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c00dcb0 con 0x55e4e8069270 2024-03-28T18:43:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.163+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 0x55e4e807ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.164+0000 7f0594f7b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 0x55e4e807ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:43094/0 (socket says 172.21.15.145:43094) 2024-03-28T18:43:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.164+0000 7f0594f7b640 1 -- 172.21.15.145:0/128350258 learned_addr learned my addr 172.21.15.145:0/128350258 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.164+0000 7f0599dca740 10 client.0 _ll_register_callbacks cb 0x7f058003eb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.165+0000 7f059577c640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 0x55e4e807e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.167+0000 7f058ffff640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e807d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.168+0000 7f059577c640 1 -- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 msgr2=0x55e4e807ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.168+0000 7f059577c640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 0x55e4e807ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.168+0000 7f059577c640 1 -- 172.21.15.145:0/128350258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 msgr2=0x55e4e807d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.168+0000 7f059577c640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e807d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.168+0000 7f059577c640 1 -- 172.21.15.145:0/128350258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4e807e770 con 0x55e4e8073480 2024-03-28T18:43:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.169+0000 7f059577c640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 0x55e4e807e180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f058809c6f0 tx=0x7f058809c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b700dd12576327c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.172+0000 7f056ffff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:43:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.172+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05880b5060 con 0x55e4e8073480 2024-03-28T18:43:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.172+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f05880cf2b0 con 0x55e4e8073480 2024-03-28T18:43:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.172+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0584018300 con 0x55e4e8073480 2024-03-28T18:43:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.176+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05880ba050 con 0x55e4e8073480 2024-03-28T18:43:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.177+0000 7f0599dca740 10 client.25583 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.177+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e4e80df340 con 0x55e4e8073480 2024-03-28T18:43:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.177+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0584018300 con 0x55e4e8073480 2024-03-28T18:43:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.178+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f05880aa080 con 0x55e4e8073480 2024-03-28T18:43:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 10 client.25583 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 20 client.25583 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80dfc10 0x55e4e80e2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 4 client.25583 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:43:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.179+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e4e7fda3a0 con 0x55e4e80dfc10 2024-03-28T18:43:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.184+0000 7f0594f7b640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80dfc10 0x55e4e80e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.192+0000 7f0594f7b640 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80dfc10 0x55e4e80e2030 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffabdc43d968500c server_cookie=a226101dd942748a in_seq=0 out_seq=0 2024-03-28T18:43:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:32.192+0000 7f056ffff640 10 client.25583 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:33.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:32 smithi192 ceph-mon[36589]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.21k op/s 2024-03-28T18:43:33.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:32 smithi164 ceph-mon[36248]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.21k op/s 2024-03-28T18:43:33.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:32 smithi145 ceph-mon[32151]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 727 MiB/s wr, 1.21k op/s 2024-03-28T18:43:35.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:34 smithi192 ceph-mon[36589]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 691 MiB/s wr, 1.18k op/s 2024-03-28T18:43:35.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:34 smithi164 ceph-mon[36248]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 691 MiB/s wr, 1.18k op/s 2024-03-28T18:43:35.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:34 smithi145 ceph-mon[32151]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 691 MiB/s wr, 1.18k op/s 2024-03-28T18:43:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:36 smithi192 ceph-mon[36589]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 870 MiB/s wr, 1.48k op/s 2024-03-28T18:43:37.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:36 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:37.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:36 smithi164 ceph-mon[36248]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 870 MiB/s wr, 1.48k op/s 2024-03-28T18:43:37.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:36 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:37.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:36 smithi145 ceph-mon[32151]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 870 MiB/s wr, 1.48k op/s 2024-03-28T18:43:37.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:36 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.966+0000 7f056ffff640 1 -- 172.21.15.145:0/128350258 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e4e7fda3a0 con 0x55e4e80dfc10 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.966+0000 7f056ffff640 10 client.25583 handle_command_reply: tid=0 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.966+0000 7f0599dca740 1 client.25583 shutdown 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 msgr2=0x55e4e807e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 0x55e4e807e180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f058809c6f0 tx=0x7f058809c1c0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 shutdown_connections 2024-03-28T18:43:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e4e80dfc10 0x55e4e80e2030 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e4e8073480 0x55e4e807e180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4e806e310 0x55e4e807d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 --2- 172.21.15.145:0/128350258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4e8069270 0x55e4e807ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 >> 172.21.15.145:0/128350258 conn(0x55e4e8068e90 msgr2=0x55e4e806c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f056ffff640 0 client.25583 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.967+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 shutdown_connections 2024-03-28T18:43:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.968+0000 7f0599dca740 1 -- 172.21.15.145:0/128350258 wait complete. 2024-03-28T18:43:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.968+0000 7f0599dca740 20 client.25583 trim_cache size 0 max 16384 2024-03-28T18:43:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:43:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:43:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:43:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:43:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.972+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 msgr2=0x7f05780a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.972+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 0x7f05780a4ec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0580004880 tx=0x7f0580005850 comp rx=0 tx=0).stop 2024-03-28T18:43:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.972+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 msgr2=0x7f05901becc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.972+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f058800cad0 tx=0x7f058800cf90 comp rx=0 tx=0).stop 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 shutdown_connections 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f05780a2a00 0x7f05780a4ec0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0590101050 0x7f05901becc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05901006b0 0x7f0590195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 --2- 172.21.15.145:0/675095680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05900ff580 0x7f05901956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.973+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 >> 172.21.15.145:0/675095680 conn(0x7f05900fae80 msgr2=0x7f0590103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.977+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 shutdown_connections 2024-03-28T18:43:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:37.979+0000 7f05971df640 1 -- 172.21.15.145:0/675095680 wait complete. 2024-03-28T18:43:39.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:38 smithi192 ceph-mon[36589]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.18k op/s 2024-03-28T18:43:39.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:38 smithi164 ceph-mon[36248]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.18k op/s 2024-03-28T18:43:39.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:38 smithi145 ceph-mon[32151]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 636 MiB/s wr, 1.18k op/s 2024-03-28T18:43:41.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:40 smithi192 ceph-mon[36589]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.38k op/s 2024-03-28T18:43:41.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:40 smithi164 ceph-mon[36248]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.38k op/s 2024-03-28T18:43:41.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:40 smithi145 ceph-mon[32151]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.38k op/s 2024-03-28T18:43:43.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:42 smithi192 ceph-mon[36589]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 756 MiB/s wr, 1.39k op/s 2024-03-28T18:43:43.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:42 smithi164 ceph-mon[36248]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 756 MiB/s wr, 1.39k op/s 2024-03-28T18:43:43.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:42 smithi145 ceph-mon[32151]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 756 MiB/s wr, 1.39k op/s 2024-03-28T18:43:45.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:44 smithi164 ceph-mon[36248]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 627 MiB/s wr, 1.11k op/s 2024-03-28T18:43:45.280 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:44 smithi145 ceph-mon[32151]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 627 MiB/s wr, 1.11k op/s 2024-03-28T18:43:45.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:44 smithi192 ceph-mon[36589]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 627 MiB/s wr, 1.11k op/s 2024-03-28T18:43:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:46 smithi164 ceph-mon[36248]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 800 MiB/s wr, 1.64k op/s 2024-03-28T18:43:47.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:46 smithi145 ceph-mon[32151]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 800 MiB/s wr, 1.64k op/s 2024-03-28T18:43:47.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:46 smithi192 ceph-mon[36589]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 800 MiB/s wr, 1.64k op/s 2024-03-28T18:43:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:48 smithi164 ceph-mon[36248]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 611 MiB/s wr, 1.41k op/s 2024-03-28T18:43:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:48 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:49.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:48 smithi145 ceph-mon[32151]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 611 MiB/s wr, 1.41k op/s 2024-03-28T18:43:49.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:49.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:48 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:49.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:48 smithi192 ceph-mon[36589]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 611 MiB/s wr, 1.41k op/s 2024-03-28T18:43:49.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:49.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:48 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:50.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:43:50.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:43:51.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:51 smithi145 ceph-mon[32151]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.66k op/s 2024-03-28T18:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.339+0000 7f4251d72640 1 -- 172.21.15.145:0/4024844265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c0732b0 msgr2=0x7f424c108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.339+0000 7f4251d72640 1 --2- 172.21.15.145:0/4024844265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c0732b0 0x7f424c108cd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4240009900 tx=0x7f424002bc80 comp rx=0 tx=0).stop 2024-03-28T18:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.340+0000 7f4251d72640 1 -- 172.21.15.145:0/4024844265 shutdown_connections 2024-03-28T18:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.340+0000 7f4251d72640 1 --2- 172.21.15.145:0/4024844265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c109390 0x7f424c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.340+0000 7f4251d72640 1 --2- 172.21.15.145:0/4024844265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c0732b0 0x7f424c108cd0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4240009900 tx=0x7f424002bc80 comp rx=0 tx=0).stop 2024-03-28T18:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.340+0000 7f4251d72640 1 --2- 172.21.15.145:0/4024844265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.340+0000 7f4251d72640 1 -- 172.21.15.145:0/4024844265 >> 172.21.15.145:0/4024844265 conn(0x7f424c06e510 msgr2=0x7f424c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.343+0000 7f4251d72640 1 -- 172.21.15.145:0/4024844265 shutdown_connections 2024-03-28T18:43:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.347+0000 7f4251d72640 1 -- 172.21.15.145:0/4024844265 wait complete. 2024-03-28T18:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.348+0000 7f4251d72640 1 Processor -- start 2024-03-28T18:43:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.349+0000 7f4251d72640 1 -- start start 2024-03-28T18:43:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c109390 0x7f424c115490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c1b4710 0x7f424c1b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c11c010 con 0x7f424c1b4710 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c11be90 con 0x7f424c109390 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4251d72640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c11c190 con 0x7f424c072950 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.350+0000 7f4250d70640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57402/0 (socket says 172.21.15.145:57402) 2024-03-28T18:43:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 -- 172.21.15.145:0/3377761646 learned_addr learned my addr 172.21.15.145:0/3377761646 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 -- 172.21.15.145:0/3377761646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c109390 msgr2=0x7f424c115490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c109390 0x7f424c115490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 -- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c1b4710 msgr2=0x7f424c1b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c1b4710 0x7f424c1b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.351+0000 7f4250d70640 1 -- 172.21.15.145:0/3377761646 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c1b7070 con 0x7f424c072950 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.352+0000 7f4250d70640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f424400d950 tx=0x7f424400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.353+0000 7f4249ffb640 1 -- 172.21.15.145:0/3377761646 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244017070 con 0x7f424c072950 2024-03-28T18:43:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.353+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c1b7360 con 0x7f424c072950 2024-03-28T18:43:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.353+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c1b78b0 con 0x7f424c072950 2024-03-28T18:43:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.353+0000 7f4249ffb640 1 -- 172.21.15.145:0/3377761646 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4244009c30 con 0x7f424c072950 2024-03-28T18:43:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.353+0000 7f4249ffb640 1 -- 172.21.15.145:0/3377761646 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424400f690 con 0x7f424c072950 2024-03-28T18:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.359+0000 7f4255bc0740 1 Processor -- start 2024-03-28T18:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.359+0000 7f4255bc0740 1 -- start start 2024-03-28T18:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.360+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.361+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.361+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.361+0000 7f4255bc0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a92728b10 con 0x555a927c3b20 2024-03-28T18:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.361+0000 7f4255bc0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a926d1900 con 0x555a927c9680 2024-03-28T18:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.361+0000 7f4255bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a92724bb0 con 0x555a927c4510 2024-03-28T18:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.362+0000 7f424bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.362+0000 7f424bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:59998/0 (socket says 172.21.15.145:59998) 2024-03-28T18:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.362+0000 7f424bfff640 1 -- 172.21.15.145:0/2749935634 learned_addr learned my addr 172.21.15.145:0/2749935634 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.362+0000 7f4250d70640 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 -- 172.21.15.145:0/2749935634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 msgr2=0x555a927c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 -- 172.21.15.145:0/2749935634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 msgr2=0x555a927c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 -- 172.21.15.145:0/2749935634 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a9243d210 con 0x555a927c9680 2024-03-28T18:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f4249ffb640 1 -- 172.21.15.145:0/3377761646 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f424400f830 con 0x7f424c072950 2024-03-28T18:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.363+0000 7f424bfff640 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4240009230 tx=0x7f4240005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15e38405c05393d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.364+0000 7f4249ffb640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 0x7f42340a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.364+0000 7f4249ffb640 1 -- 172.21.15.145:0/3377761646 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f42440ce6a0 con 0x7f424c072950 2024-03-28T18:43:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.364+0000 7f4232ffd640 1 -- 172.21.15.145:0/2749935634 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240039040 con 0x555a927c9680 2024-03-28T18:43:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.364+0000 7f4232ffd640 1 -- 172.21.15.145:0/2749935634 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4240004940 con 0x555a927c9680 2024-03-28T18:43:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.364+0000 7f4232ffd640 1 -- 172.21.15.145:0/2749935634 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240004d80 con 0x555a927c9680 2024-03-28T18:43:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.366+0000 7f4251571640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 0x7f42340a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.366+0000 7f4255bc0740 1 -- 172.21.15.145:0/2749935634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 msgr2=0x555a927cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.366+0000 7f4255bc0740 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4240009230 tx=0x7f4240005e80 comp rx=0 tx=0).stop 2024-03-28T18:43:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.370+0000 7f4255bc0740 1 -- 172.21.15.145:0/2749935634 shutdown_connections 2024-03-28T18:43:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.370+0000 7f4255bc0740 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927cbb10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.370+0000 7f4255bc0740 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.370+0000 7f4255bc0740 1 --2- 172.21.15.145:0/2749935634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.370+0000 7f4255bc0740 1 -- 172.21.15.145:0/2749935634 >> 172.21.15.145:0/2749935634 conn(0x555a927bf090 msgr2=0x555a927bf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.371+0000 7f4255bc0740 1 -- 172.21.15.145:0/2749935634 shutdown_connections 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.371+0000 7f4255bc0740 1 -- 172.21.15.145:0/2749935634 wait complete. 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.371+0000 7f4255bc0740 1 -- start start 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.371+0000 7f4255bc0740 1 Processor -- start 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c3b20 0x555a927cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a927cc570 con 0x555a927c9680 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a926e4d50 con 0x555a927c4510 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a927cc6f0 con 0x555a927c3b20 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.372+0000 7f4255bc0740 10 client.0 _ll_register_callbacks cb 0x555a926e4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.373+0000 7f424bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.373+0000 7f424bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:60010/0 (socket says 172.21.15.145:60010) 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.373+0000 7f424bfff640 1 -- 172.21.15.145:0/1268859316 learned_addr learned my addr 172.21.15.145:0/1268859316 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.373+0000 7f4250d70640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c3b20 0x555a927cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 -- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c3b20 msgr2=0x555a927cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c3b20 0x555a927cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 -- 172.21.15.145:0/1268859316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 msgr2=0x555a927d52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 -- 172.21.15.145:0/1268859316 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a92509620 con 0x555a927c4510 2024-03-28T18:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.374+0000 7f424bfff640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4240004b10 tx=0x7f42400048a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b1e9f08223bbfd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.376+0000 7f422bfff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:43:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.376+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424004a060 con 0x555a927c4510 2024-03-28T18:43:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.376+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4240004940 con 0x555a927c4510 2024-03-28T18:43:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.376+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f4240005210 con 0x555a927c4510 2024-03-28T18:43:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.377+0000 7f4251571640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 0x7f42340a4e80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x555a927c4d70 tx=0x7f423c009230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.379+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4240005410 con 0x555a927c4510 2024-03-28T18:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.379+0000 7f4255bc0740 10 client.25592 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.379+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555a927fa670 con 0x555a927c4510 2024-03-28T18:43:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.385+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4240004940 con 0x555a927c4510 2024-03-28T18:43:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f424002cce0 con 0x555a927c4510 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 10 client.25592 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 10 client.25592 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 20 client.25592 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a927fb9c0 0x555a927fddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 4 client.25592 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.386+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a92736050 con 0x555a927fb9c0 2024-03-28T18:43:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.388+0000 7f4250d70640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a927fb9c0 0x555a927fddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.388+0000 7f4250d70640 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a927fb9c0 0x555a927fddb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1397489d13e3aa21 server_cookie=d791e7d9f9d500a0 in_seq=0 out_seq=0 2024-03-28T18:43:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.388+0000 7f422bfff640 10 client.25592 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.390+0000 7f422bfff640 1 -- 172.21.15.145:0/1268859316 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a92736050 con 0x555a927fb9c0 2024-03-28T18:43:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.390+0000 7f422bfff640 10 client.25592 handle_command_reply: tid=0 2024-03-28T18:43:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.394+0000 7f4255bc0740 1 client.25592 shutdown 2024-03-28T18:43:51.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:51 smithi192 ceph-mon[36589]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.66k op/s 2024-03-28T18:43:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.406+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 msgr2=0x555a927ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.406+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4240004b10 tx=0x7f42400048a0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 shutdown_connections 2024-03-28T18:43:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a927fb9c0 0x555a927fddb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c4510 0x555a927ce2b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1268859316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c3b20 0x555a927cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f422bfff640 0 client.25592 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.414+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 >> 172.21.15.145:0/1268859316 conn(0x555a927bf090 msgr2=0x555a927be290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.443+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 shutdown_connections 2024-03-28T18:43:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.445+0000 7f4255bc0740 1 -- 172.21.15.145:0/1268859316 wait complete. 2024-03-28T18:43:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.446+0000 7f4255bc0740 20 client.25592 trim_cache size 0 max 16384 2024-03-28T18:43:51.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:51 smithi164 ceph-mon[36248]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 725 MiB/s wr, 1.66k op/s 2024-03-28T18:43:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.490+0000 7f4255bc0740 1 Processor -- start 2024-03-28T18:43:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.490+0000 7f4255bc0740 1 -- start start 2024-03-28T18:43:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423800f960 con 0x555a927c3b20 2024-03-28T18:43:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380112e0 con 0x555a927c9680 2024-03-28T18:43:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.491+0000 7f4255bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423800ffc0 con 0x555a927c4510 2024-03-28T18:43:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.493+0000 7f4250d70640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.493+0000 7f4250d70640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41274/0 (socket says 172.21.15.145:41274) 2024-03-28T18:43:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.493+0000 7f4250d70640 1 -- 172.21.15.145:0/1688305392 learned_addr learned my addr 172.21.15.145:0/1688305392 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.493+0000 7f4251571640 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f424bfff640 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f4250d70640 1 -- 172.21.15.145:0/1688305392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 msgr2=0x555a927d52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f4250d70640 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f4250d70640 1 -- 172.21.15.145:0/1688305392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 msgr2=0x555a927c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f4250d70640 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.498+0000 7f4250d70640 1 -- 172.21.15.145:0/1688305392 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a927fe340 con 0x555a927c3b20 2024-03-28T18:43:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.507+0000 7f4250d70640 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4244097600 tx=0x7f4244000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9401fe7d4dd02c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.511+0000 7f42327fc640 1 -- 172.21.15.145:0/1688305392 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42440b3070 con 0x555a927c3b20 2024-03-28T18:43:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.511+0000 7f42327fc640 1 -- 172.21.15.145:0/1688305392 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42440a33d0 con 0x555a927c3b20 2024-03-28T18:43:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.511+0000 7f42327fc640 1 -- 172.21.15.145:0/1688305392 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244020040 con 0x555a927c3b20 2024-03-28T18:43:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.515+0000 7f4255bc0740 1 -- 172.21.15.145:0/1688305392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 msgr2=0x555a927c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.515+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4244097600 tx=0x7f4244000bc0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.519+0000 7f4255bc0740 1 -- 172.21.15.145:0/1688305392 shutdown_connections 2024-03-28T18:43:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.519+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c9680 0x555a927c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.519+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.519+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1688305392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c3b20 0x555a927c3f00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.519+0000 7f4255bc0740 1 -- 172.21.15.145:0/1688305392 >> 172.21.15.145:0/1688305392 conn(0x555a927bf090 msgr2=0x555a927c8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.523+0000 7f4255bc0740 1 -- 172.21.15.145:0/1688305392 shutdown_connections 2024-03-28T18:43:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.524+0000 7f4255bc0740 1 -- 172.21.15.145:0/1688305392 wait complete. 2024-03-28T18:43:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.525+0000 7f4255bc0740 1 -- start start 2024-03-28T18:43:51.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.525+0000 7f4255bc0740 1 Processor -- start 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c3b20 0x555a927d2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4238010c80 con 0x555a927c9680 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4238010620 con 0x555a927c3b20 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.526+0000 7f4255bc0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423800dbd0 con 0x555a927c4510 2024-03-28T18:43:51.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.527+0000 7f4255bc0740 10 client.0 _ll_register_callbacks cb 0x7f4240004740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.527+0000 7f4251571640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.528+0000 7f4251571640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41282/0 (socket says 172.21.15.145:41282) 2024-03-28T18:43:51.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.528+0000 7f4251571640 1 -- 172.21.15.145:0/1949368113 learned_addr learned my addr 172.21.15.145:0/1949368113 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:43:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.531+0000 7f424bfff640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.531 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.531+0000 7f4250d70640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c3b20 0x555a927d2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.531+0000 7f424bfff640 1 -- 172.21.15.145:0/1949368113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c3b20 msgr2=0x555a927d2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.532+0000 7f424bfff640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c3b20 0x555a927d2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.532+0000 7f424bfff640 1 -- 172.21.15.145:0/1949368113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 msgr2=0x555a927d4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.532+0000 7f424bfff640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.532+0000 7f424bfff640 1 -- 172.21.15.145:0/1949368113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a927d4810 con 0x555a927c4510 2024-03-28T18:43:51.533 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.533+0000 7f424bfff640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d3260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f424003d4e0 tx=0x7f4240005220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c24bd6e7f8e82eda server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.534+0000 7f422bfff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:43:51.534 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.534+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424004a070 con 0x555a927c4510 2024-03-28T18:43:51.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.534+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f424002c9a0 con 0x555a927c4510 2024-03-28T18:43:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.535+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42440a33d0 con 0x555a927c4510 2024-03-28T18:43:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.537+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4240054080 con 0x555a927c4510 2024-03-28T18:43:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.541+0000 7f4255bc0740 10 client.25678 fetch_fsmap learned FSMap version 31 2024-03-28T18:43:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.541+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555a92857460 con 0x555a927c4510 2024-03-28T18:43:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42440a33d0 con 0x555a927c4510 2024-03-28T18:43:51.546 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f424003c070 con 0x555a927c4510 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 10 client.25678 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 20 client.25678 populate_metadata read hostname 'smithi145' 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a92857d80 0x555a9285a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 4 client.25678 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:43:51.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.546+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555a927303a0 con 0x555a92857d80 2024-03-28T18:43:51.553 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.552+0000 7f4250d70640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a92857d80 0x555a9285a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.562+0000 7f4250d70640 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a92857d80 0x555a9285a1a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87b70b8265cf71fa server_cookie=85c3239e05a47a6c in_seq=0 out_seq=0 2024-03-28T18:43:51.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:51.564+0000 7f422bfff640 10 client.25678 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:53 smithi192 ceph-mon[36589]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 686 MiB/s wr, 1.57k op/s 2024-03-28T18:43:53.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:53 smithi164 ceph-mon[36248]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 686 MiB/s wr, 1.57k op/s 2024-03-28T18:43:53.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:53 smithi145 ceph-mon[32151]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 686 MiB/s wr, 1.57k op/s 2024-03-28T18:43:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:55 smithi192 ceph-mon[36589]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 MiB/s wr, 1.31k op/s 2024-03-28T18:43:55.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:55 smithi164 ceph-mon[36248]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 MiB/s wr, 1.31k op/s 2024-03-28T18:43:55.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:55 smithi145 ceph-mon[32151]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 MiB/s wr, 1.31k op/s 2024-03-28T18:43:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.239+0000 7f422bfff640 1 -- 172.21.15.145:0/1949368113 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555a927303a0 con 0x555a92857d80 2024-03-28T18:43:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.239+0000 7f422bfff640 10 client.25678 handle_command_reply: tid=0 2024-03-28T18:43:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.239+0000 7f4255bc0740 1 client.25678 shutdown 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 msgr2=0x555a927d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d3260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f424003d4e0 tx=0x7f4240005220 comp rx=0 tx=0).stop 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 shutdown_connections 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555a92857d80 0x555a9285a1a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555a927c9680 0x555a927d4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555a927c4510 0x555a927d3260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 --2- 172.21.15.145:0/1949368113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555a927c3b20 0x555a927d2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.240+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 >> 172.21.15.145:0/1949368113 conn(0x555a927bf090 msgr2=0x555a927c22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.241+0000 7f422bfff640 0 client.25678 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:43:56.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.241+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 shutdown_connections 2024-03-28T18:43:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.241+0000 7f4255bc0740 1 -- 172.21.15.145:0/1949368113 wait complete. 2024-03-28T18:43:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.241+0000 7f4255bc0740 20 client.25678 trim_cache size 0 max 16384 2024-03-28T18:43:56.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:43:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:43:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:43:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:43:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.242+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 msgr2=0x7f42340a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.242+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 0x7f42340a4e80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x555a927c4d70 tx=0x7f423c009230 comp rx=0 tx=0).stop 2024-03-28T18:43:56.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.242+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 msgr2=0x7f424c114f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.242+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f424400d950 tx=0x7f424400de10 comp rx=0 tx=0).stop 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 shutdown_connections 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c1b4710 0x7f424c1b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f424c109390 0x7f424c115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f42340a29c0 0x7f42340a4e80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 --2- 172.21.15.145:0/3377761646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f424c072950 0x7f424c114f50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 >> 172.21.15.145:0/3377761646 conn(0x7f424c06e510 msgr2=0x7f424c06ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 shutdown_connections 2024-03-28T18:43:56.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:43:56.244+0000 7f4251d72640 1 -- 172.21.15.145:0/3377761646 wait complete. 2024-03-28T18:43:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:57 smithi192 ceph-mon[36589]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 764 MiB/s wr, 1.52k op/s 2024-03-28T18:43:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:57 smithi164 ceph-mon[36248]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 764 MiB/s wr, 1.52k op/s 2024-03-28T18:43:57.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:57 smithi145 ceph-mon[32151]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 764 MiB/s wr, 1.52k op/s 2024-03-28T18:43:59.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:43:59 smithi192 ceph-mon[36589]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 647 MiB/s wr, 1.10k op/s 2024-03-28T18:43:59.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:43:59 smithi164 ceph-mon[36248]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 647 MiB/s wr, 1.10k op/s 2024-03-28T18:43:59.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:43:59 smithi145 ceph-mon[32151]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 647 MiB/s wr, 1.10k op/s 2024-03-28T18:44:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:00 smithi164 ceph-mon[36248]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.30k op/s 2024-03-28T18:44:00.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:00 smithi145 ceph-mon[32151]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.30k op/s 2024-03-28T18:44:00.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:00 smithi192 ceph-mon[36589]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.30k op/s 2024-03-28T18:44:03.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:02 smithi192 ceph-mon[36589]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.44k op/s 2024-03-28T18:44:03.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:02 smithi164 ceph-mon[36248]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.44k op/s 2024-03-28T18:44:03.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:02 smithi145 ceph-mon[32151]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 741 MiB/s wr, 1.44k op/s 2024-03-28T18:44:05.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:04 smithi192 ceph-mon[36589]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 630 MiB/s wr, 1.22k op/s 2024-03-28T18:44:05.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:04 smithi164 ceph-mon[36248]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 630 MiB/s wr, 1.22k op/s 2024-03-28T18:44:05.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:04 smithi145 ceph-mon[32151]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 630 MiB/s wr, 1.22k op/s 2024-03-28T18:44:07.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:06 smithi192 ceph-mon[36589]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 MiB/s wr, 1.46k op/s 2024-03-28T18:44:07.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:06 smithi164 ceph-mon[36248]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 MiB/s wr, 1.46k op/s 2024-03-28T18:44:07.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:06 smithi145 ceph-mon[32151]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 MiB/s wr, 1.46k op/s 2024-03-28T18:44:09.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:08 smithi192 ceph-mon[36589]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.47k op/s 2024-03-28T18:44:09.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:08 smithi164 ceph-mon[36248]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.47k op/s 2024-03-28T18:44:09.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:08 smithi145 ceph-mon[32151]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 719 MiB/s wr, 1.47k op/s 2024-03-28T18:44:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.569+0000 7f191e590640 1 -- 172.21.15.145:0/3817578736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1918108910 msgr2=0x7f191810ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.569+0000 7f191e590640 1 --2- 172.21.15.145:0/3817578736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1918108910 0x7f191810ad00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f190c009860 tx=0x7f190c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:44:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.572+0000 7f191e590640 1 -- 172.21.15.145:0/3817578736 shutdown_connections 2024-03-28T18:44:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.572+0000 7f191e590640 1 --2- 172.21.15.145:0/3817578736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f191810dd70 0x7f1918110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.572+0000 7f191e590640 1 --2- 172.21.15.145:0/3817578736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f191810b2d0 0x7f191810d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.572+0000 7f191e590640 1 --2- 172.21.15.145:0/3817578736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1918108910 0x7f191810ad00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.572+0000 7f191e590640 1 -- 172.21.15.145:0/3817578736 >> 172.21.15.145:0/3817578736 conn(0x7f191806e8a0 msgr2=0x7f1918070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.576+0000 7f191e590640 1 -- 172.21.15.145:0/3817578736 shutdown_connections 2024-03-28T18:44:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.578+0000 7f191e590640 1 -- 172.21.15.145:0/3817578736 wait complete. 2024-03-28T18:44:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.579+0000 7f191e590640 1 Processor -- start 2024-03-28T18:44:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.579+0000 7f191e590640 1 -- start start 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f191810b2d0 0x7f19181a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f191810dd70 0x7f19181add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181139a0 con 0x7f191810dd70 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1918113820 con 0x7f191810b2d0 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f191e590640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1918113b20 con 0x7f1918108910 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f1917fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.580+0000 7f1917fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:39458/0 (socket says 172.21.15.145:39458) 2024-03-28T18:44:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.581+0000 7f1917fff640 1 -- 172.21.15.145:0/1616891615 learned_addr learned my addr 172.21.15.145:0/1616891615 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f1917fff640 1 -- 172.21.15.145:0/1616891615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f191810b2d0 msgr2=0x7f19181a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f19177fe640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f191810b2d0 0x7f19181a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f1917fff640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f191810b2d0 0x7f19181a6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f1917fff640 1 -- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f191810dd70 msgr2=0x7f19181add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f1917fff640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f191810dd70 0x7f19181add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.582+0000 7f1917fff640 1 -- 172.21.15.145:0/1616891615 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19181ae400 con 0x7f1918108910 2024-03-28T18:44:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.583+0000 7f1917fff640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f190c000c00 tx=0x7f190c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.585+0000 7f19157fa640 1 -- 172.21.15.145:0/1616891615 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c041020 con 0x7f1918108910 2024-03-28T18:44:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.585+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19181ae690 con 0x7f1918108910 2024-03-28T18:44:09.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.585+0000 7f19157fa640 1 -- 172.21.15.145:0/1616891615 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f190c0347b0 con 0x7f1918108910 2024-03-28T18:44:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.585+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19181aeb80 con 0x7f1918108910 2024-03-28T18:44:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.588+0000 7f19157fa640 1 -- 172.21.15.145:0/1616891615 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c039040 con 0x7f1918108910 2024-03-28T18:44:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.600+0000 7f192117f740 1 Processor -- start 2024-03-28T18:44:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.600+0000 7f192117f740 1 -- start start 2024-03-28T18:44:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.600+0000 7f192117f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f192117f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f192117f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f192117f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb151b10 con 0x555afb1ed320 2024-03-28T18:44:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f192117f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb0fa900 con 0x555afb1ec930 2024-03-28T18:44:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f192117f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb14dbb0 con 0x555afb1f2490 2024-03-28T18:44:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f191cb2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f191cb2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52406/0 (socket says 172.21.15.145:52406) 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f191cb2d640 1 -- 172.21.15.145:0/4096353163 learned_addr learned my addr 172.21.15.145:0/4096353163 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f19177fe640 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.601+0000 7f1917fff640 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f19157fa640 1 -- 172.21.15.145:0/1616891615 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f190c044020 con 0x7f1918108910 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 -- 172.21.15.145:0/4096353163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 msgr2=0x555afb1f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 -- 172.21.15.145:0/4096353163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 msgr2=0x555afb1f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 -- 172.21.15.145:0/4096353163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555afae66210 con 0x555afb1ec930 2024-03-28T18:44:09.603 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.602+0000 7f191cb2d640 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f190800a6c0 tx=0x7f19080091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9064297a7e300c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.604+0000 7f19157fa640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 0x7f19000a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.604 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.604+0000 7f1917fff640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 0x7f19000a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.604+0000 7f19157fa640 1 -- 172.21.15.145:0/1616891615 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f190c0f1880 con 0x7f1918108910 2024-03-28T18:44:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.605+0000 7f18fe7fc640 1 -- 172.21.15.145:0/4096353163 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908009990 con 0x555afb1ec930 2024-03-28T18:44:09.605 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.605+0000 7f18fe7fc640 1 -- 172.21.15.145:0/4096353163 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1908005500 con 0x555afb1ec930 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 -- 172.21.15.145:0/4096353163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 msgr2=0x555afb1ecd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f190800a6c0 tx=0x7f19080091e0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 -- 172.21.15.145:0/4096353163 shutdown_connections 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 --2- 172.21.15.145:0/4096353163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1ec930 0x555afb1ecd50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 -- 172.21.15.145:0/4096353163 >> 172.21.15.145:0/4096353163 conn(0x555afb1e7ea0 msgr2=0x555afb1e82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.606+0000 7f192117f740 1 -- 172.21.15.145:0/4096353163 shutdown_connections 2024-03-28T18:44:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.607+0000 7f192117f740 1 -- 172.21.15.145:0/4096353163 wait complete. 2024-03-28T18:44:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.607+0000 7f192117f740 1 -- start start 2024-03-28T18:44:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.607+0000 7f192117f740 1 Processor -- start 2024-03-28T18:44:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb1fe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb1f5490 con 0x555afb1fc0f0 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb10dd50 con 0x555afb1f2490 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555afb1f5610 con 0x555afb1ed320 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.608+0000 7f192117f740 10 client.0 _ll_register_callbacks cb 0x555afb10d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.609+0000 7f191cb2d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb1fe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.609+0000 7f19177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.610+0000 7f19177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52416/0 (socket says 172.21.15.145:52416) 2024-03-28T18:44:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.610+0000 7f19177fe640 1 -- 172.21.15.145:0/2031846651 learned_addr learned my addr 172.21.15.145:0/2031846651 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.610+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f1917fff640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 0x7f19000a4fa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f190c009830 tx=0x7f190c0f7530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f191cb2d640 1 -- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 msgr2=0x555afb1f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f191cb2d640 1 -- 172.21.15.145:0/2031846651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 msgr2=0x555afb1f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.611+0000 7f191cb2d640 1 -- 172.21.15.145:0/2031846651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555afaf32620 con 0x555afb1fc0f0 2024-03-28T18:44:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.612+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb1fe4e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19080096e0 tx=0x7f1908004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86849d4e3ed6b812 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.613+0000 7f18f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:44:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.613+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908005000 con 0x555afb1fc0f0 2024-03-28T18:44:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.613+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555afb15f050 con 0x555afb1fc0f0 2024-03-28T18:44:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.613+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f19080051a0 con 0x555afb1fc0f0 2024-03-28T18:44:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.619+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19080053a0 con 0x555afb1fc0f0 2024-03-28T18:44:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.620+0000 7f192117f740 10 client.15822 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.620+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555afb212940 con 0x555afb1fc0f0 2024-03-28T18:44:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.633+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555afb15f050 con 0x555afb1fc0f0 2024-03-28T18:44:09.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.633+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1908013070 con 0x555afb1fc0f0 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 10 client.15822 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 20 client.15822 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb213df0 0x555afb2161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 4 client.15822 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.635+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555afb1593a0 con 0x555afb213df0 2024-03-28T18:44:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.638+0000 7f19177fe640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb213df0 0x555afb2161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.652+0000 7f18f77fe640 10 client.15822 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.652+0000 7f19177fe640 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb213df0 0x555afb2161e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33ef4b6176bf6741 server_cookie=6bde823a0af709a8 in_seq=0 out_seq=0 2024-03-28T18:44:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.662+0000 7f18f77fe640 1 -- 172.21.15.145:0/2031846651 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555afb1593a0 con 0x555afb213df0 2024-03-28T18:44:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.662+0000 7f18f77fe640 10 client.15822 handle_command_reply: tid=0 2024-03-28T18:44:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.663+0000 7f192117f740 1 client.15822 shutdown 2024-03-28T18:44:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.670+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 msgr2=0x555afb1fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.670+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb1fe4e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19080096e0 tx=0x7f1908004640 comp rx=0 tx=0).stop 2024-03-28T18:44:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 shutdown_connections 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb213df0 0x555afb2161e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb1fe4e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 --2- 172.21.15.145:0/2031846651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 >> 172.21.15.145:0/2031846651 conn(0x555afb1e7ea0 msgr2=0x555afb1e6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.672+0000 7f18f77fe640 0 client.15822 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.675+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 shutdown_connections 2024-03-28T18:44:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.677+0000 7f192117f740 1 -- 172.21.15.145:0/2031846651 wait complete. 2024-03-28T18:44:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.680+0000 7f192117f740 20 client.15822 trim_cache size 0 max 16384 2024-03-28T18:44:09.727 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.727+0000 7f192117f740 1 Processor -- start 2024-03-28T18:44:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.728+0000 7f192117f740 1 -- start start 2024-03-28T18:44:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.728+0000 7f192117f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.728+0000 7f192117f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f192117f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f192117f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181b0800 con 0x555afb1ed320 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f192117f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181b2180 con 0x555afb1fc0f0 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f191cb2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f191cb2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52420/0 (socket says 172.21.15.145:52420) 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f191cb2d640 1 -- 172.21.15.145:0/2517846962 learned_addr learned my addr 172.21.15.145:0/2517846962 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.729+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181b0e60 con 0x555afb1f2490 2024-03-28T18:44:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 -- 172.21.15.145:0/2517846962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 msgr2=0x555afb1f70c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:44:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 -- 172.21.15.145:0/2517846962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 msgr2=0x555afb1f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 -- 172.21.15.145:0/2517846962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555afb216810 con 0x555afb1fc0f0 2024-03-28T18:44:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.730+0000 7f191cb2d640 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x555afb1f5de0 tx=0x7f1908001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcef8533ae68ec06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.740+0000 7f18fdffb640 1 -- 172.21.15.145:0/2517846962 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908005540 con 0x555afb1fc0f0 2024-03-28T18:44:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.740+0000 7f18fdffb640 1 -- 172.21.15.145:0/2517846962 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1908004f70 con 0x555afb1fc0f0 2024-03-28T18:44:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.740+0000 7f18fdffb640 1 -- 172.21.15.145:0/2517846962 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19080218c0 con 0x555afb1fc0f0 2024-03-28T18:44:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 msgr2=0x555afb1eb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x555afb1f5de0 tx=0x7f1908001e50 comp rx=0 tx=0).stop 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 shutdown_connections 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1fc0f0 0x555afb1eb100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1f2490 0x555afb1f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 --2- 172.21.15.145:0/2517846962 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1ed320 0x555afb1f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.742+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 >> 172.21.15.145:0/2517846962 conn(0x555afb1e7ea0 msgr2=0x555afb1e8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.743+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 shutdown_connections 2024-03-28T18:44:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.743+0000 7f192117f740 1 -- 172.21.15.145:0/2517846962 wait complete. 2024-03-28T18:44:09.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.743+0000 7f192117f740 1 -- start start 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 Processor -- start 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb223c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb225070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181b1b20 con 0x555afb1fc0f0 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181b14c0 con 0x555afb1f2490 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181a5740 con 0x555afb1ed320 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.744+0000 7f192117f740 10 client.0 _ll_register_callbacks cb 0x7f190800a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.745+0000 7f19177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.745+0000 7f19177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52430/0 (socket says 172.21.15.145:52430) 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.745+0000 7f19177fe640 1 -- 172.21.15.145:0/3491197990 learned_addr learned my addr 172.21.15.145:0/3491197990 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.745+0000 7f191cb2d640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb225070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.745+0000 7f191cb2d640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb223c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 -- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 msgr2=0x555afb223c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb223c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 -- 172.21.15.145:0/3491197990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 msgr2=0x555afb225070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:09.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb225070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 -- 172.21.15.145:0/3491197990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555afb225660 con 0x555afb1f2490 2024-03-28T18:44:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.746+0000 7f19177fe640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1910012560 tx=0x7f191001eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed2e66874e0d439 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.747+0000 7f18f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:44:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.747+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910004d20 con 0x555afb1f2490 2024-03-28T18:44:09.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.747+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f19100140d0 con 0x555afb1f2490 2024-03-28T18:44:09.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.750+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1908004f70 con 0x555afb1f2490 2024-03-28T18:44:09.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.755+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19100142d0 con 0x555afb1f2490 2024-03-28T18:44:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.758+0000 7f192117f740 10 client.25613 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.758+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555afb26ebe0 con 0x555afb1f2490 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1908004f70 con 0x555afb1f2490 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f19100129d0 con 0x555afb1f2490 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 10 client.25613 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 20 client.25613 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb26f4c0 0x555afb2718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:09.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 4 client.25613 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:44:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.763+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555afb0d7710 con 0x555afb26f4c0 2024-03-28T18:44:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.764+0000 7f191cb2d640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb26f4c0 0x555afb2718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.765+0000 7f191cb2d640 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb26f4c0 0x555afb2718e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb40470684f0af3c server_cookie=c9de36b8103f58be in_seq=0 out_seq=0 2024-03-28T18:44:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:09.765+0000 7f18f77fe640 10 client.25613 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:11.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:10 smithi192 ceph-mon[36589]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 713 MiB/s wr, 1.44k op/s 2024-03-28T18:44:11.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:10 smithi164 ceph-mon[36248]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 713 MiB/s wr, 1.44k op/s 2024-03-28T18:44:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:10 smithi145 ceph-mon[32151]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 713 MiB/s wr, 1.44k op/s 2024-03-28T18:44:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.310+0000 7f18f77fe640 1 -- 172.21.15.145:0/3491197990 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555afb0d7710 con 0x555afb26f4c0 2024-03-28T18:44:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.310+0000 7f18f77fe640 10 client.25613 handle_command_reply: tid=0 2024-03-28T18:44:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.310+0000 7f192117f740 1 client.25613 shutdown 2024-03-28T18:44:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 msgr2=0x555afb224190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1910012560 tx=0x7f191001eac0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 shutdown_connections 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x555afb26f4c0 0x555afb2718e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555afb1fc0f0 0x555afb225070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x555afb1f2490 0x555afb224190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 --2- 172.21.15.145:0/3491197990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555afb1ed320 0x555afb223c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 >> 172.21.15.145:0/3491197990 conn(0x555afb1e7ea0 msgr2=0x555afb1f6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f18f77fe640 0 client.25613 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 shutdown_connections 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 1 -- 172.21.15.145:0/3491197990 wait complete. 2024-03-28T18:44:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.311+0000 7f192117f740 20 client.25613 trim_cache size 0 max 16384 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.313+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 msgr2=0x7f19000a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.313+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 0x7f19000a4fa0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f190c009830 tx=0x7f190c0f7530 comp rx=0 tx=0).stop 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.313+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 msgr2=0x7f19181a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.313+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f190c000c00 tx=0x7f190c004270 comp rx=0 tx=0).stop 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 shutdown_connections 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f191810dd70 0x7f19181add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f191810b2d0 0x7f19181a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f19000a2ae0 0x7f19000a4fa0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 --2- 172.21.15.145:0/1616891615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1918108910 0x7f19181a67e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 >> 172.21.15.145:0/1616891615 conn(0x7f191806e8a0 msgr2=0x7f191810bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 shutdown_connections 2024-03-28T18:44:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:11.314+0000 7f191e590640 1 -- 172.21.15.145:0/1616891615 wait complete. 2024-03-28T18:44:13.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:12 smithi192 ceph-mon[36589]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 MiB/s wr, 1.38k op/s 2024-03-28T18:44:13.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:12 smithi164 ceph-mon[36248]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 MiB/s wr, 1.38k op/s 2024-03-28T18:44:13.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:12 smithi145 ceph-mon[32151]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 MiB/s wr, 1.38k op/s 2024-03-28T18:44:15.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:14 smithi192 ceph-mon[36589]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.04k op/s 2024-03-28T18:44:15.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:14 smithi164 ceph-mon[36248]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.04k op/s 2024-03-28T18:44:15.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:14 smithi145 ceph-mon[32151]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 638 MiB/s wr, 1.04k op/s 2024-03-28T18:44:17.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:16 smithi164 ceph-mon[36248]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T18:44:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:16 smithi145 ceph-mon[32151]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T18:44:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:16 smithi192 ceph-mon[36589]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 799 MiB/s wr, 1.36k op/s 2024-03-28T18:44:19.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:18 smithi164 ceph-mon[36248]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.33k op/s 2024-03-28T18:44:19.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:18 smithi145 ceph-mon[32151]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.33k op/s 2024-03-28T18:44:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:18 smithi192 ceph-mon[36589]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.33k op/s 2024-03-28T18:44:21.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:20 smithi164 ceph-mon[36248]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.38k op/s 2024-03-28T18:44:21.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:20 smithi145 ceph-mon[32151]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.38k op/s 2024-03-28T18:44:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:20 smithi192 ceph-mon[36589]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.38k op/s 2024-03-28T18:44:23.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:22 smithi164 ceph-mon[36248]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 MiB/s wr, 1.43k op/s 2024-03-28T18:44:23.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:22 smithi145 ceph-mon[32151]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 MiB/s wr, 1.43k op/s 2024-03-28T18:44:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:22 smithi192 ceph-mon[36589]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 762 MiB/s wr, 1.43k op/s 2024-03-28T18:44:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.638+0000 7f18f5fd4640 1 -- 172.21.15.145:0/3158435952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010d9c0 msgr2=0x7f18f010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.638+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/3158435952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010d9c0 0x7f18f010de40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009be0 tx=0x7f18e802bf30 comp rx=0 tx=0).stop 2024-03-28T18:44:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.639+0000 7f18f5fd4640 1 -- 172.21.15.145:0/3158435952 shutdown_connections 2024-03-28T18:44:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.639+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/3158435952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010d9c0 0x7f18f010de40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.639+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/3158435952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010b010 0x7f18f010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.639+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/3158435952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.639+0000 7f18f5fd4640 1 -- 172.21.15.145:0/3158435952 >> 172.21.15.145:0/3158435952 conn(0x7f18f006e6e0 msgr2=0x7f18f0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.640+0000 7f18f5fd4640 1 -- 172.21.15.145:0/3158435952 shutdown_connections 2024-03-28T18:44:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.642+0000 7f18f5fd4640 1 -- 172.21.15.145:0/3158435952 wait complete. 2024-03-28T18:44:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.643+0000 7f18f5fd4640 1 Processor -- start 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.645+0000 7f18f5fd4640 1 -- start start 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.645+0000 7f18f5fd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f0114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.645+0000 7f18f5fd4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 0x7f18f0115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18f5fd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010d9c0 0x7f18f01b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18f5fd4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f011c1f0 con 0x7f18f010d9c0 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18f5fd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f011c070 con 0x7f18f01086a0 2024-03-28T18:44:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18f5fd4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f011c370 con 0x7f18f010b010 2024-03-28T18:44:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f0114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 0x7f18f0115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f0114cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:55054/0 (socket says 172.21.15.145:55054) 2024-03-28T18:44:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18ef7fe640 1 -- 172.21.15.145:0/1261181240 learned_addr learned my addr 172.21.15.145:0/1261181240 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.646+0000 7f18effff640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010d9c0 0x7f18f01b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.648+0000 7f18eeffd640 1 -- 172.21.15.145:0/1261181240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 msgr2=0x7f18f0114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.648+0000 7f18eeffd640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f0114cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.648+0000 7f18eeffd640 1 -- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010d9c0 msgr2=0x7f18f01b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.648+0000 7f18eeffd640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010d9c0 0x7f18f01b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.648+0000 7f18eeffd640 1 -- 172.21.15.145:0/1261181240 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f01182f0 con 0x7f18f010b010 2024-03-28T18:44:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.650+0000 7f18eeffd640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 0x7f18f0115210 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f18e000d6d0 tx=0x7f18e000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.655+0000 7f18ecff9640 1 -- 172.21.15.145:0/1261181240 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0017070 con 0x7f18f010b010 2024-03-28T18:44:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.655+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f01b6e40 con 0x7f18f010b010 2024-03-28T18:44:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.655+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f01b7300 con 0x7f18f010b010 2024-03-28T18:44:24.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.659+0000 7f18f8bc1740 1 Processor -- start 2024-03-28T18:44:24.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.659+0000 7f18f8bc1740 1 -- start start 2024-03-28T18:44:24.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f5595d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 0x5561f559d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f54fab10 con 0x5561f5595920 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f54a3900 con 0x5561f559b480 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.660+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f54f6bb0 con 0x5561f5596310 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.661+0000 7f18ecff9640 1 -- 172.21.15.145:0/1261181240 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18e0010460 con 0x7f18f010b010 2024-03-28T18:44:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.661+0000 7f18ecff9640 1 -- 172.21.15.145:0/1261181240 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e000f620 con 0x7f18f010b010 2024-03-28T18:44:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.662+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.662+0000 7f18ef7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:49804/0 (socket says 172.21.15.145:49804) 2024-03-28T18:44:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.662+0000 7f18ef7fe640 1 -- 172.21.15.145:0/3715426794 learned_addr learned my addr 172.21.15.145:0/3715426794 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.664+0000 7f18eeffd640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 0x5561f559d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.664+0000 7f18ecff9640 1 -- 172.21.15.145:0/1261181240 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f18e001a070 con 0x7f18f010b010 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.665+0000 7f18effff640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f5595d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.666+0000 7f18eeffd640 1 -- 172.21.15.145:0/3715426794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 msgr2=0x5561f559ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.666+0000 7f18eeffd640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.666+0000 7f18eeffd640 1 -- 172.21.15.145:0/3715426794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 msgr2=0x5561f5595d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.666+0000 7f18eeffd640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f5595d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.666+0000 7f18eeffd640 1 -- 172.21.15.145:0/3715426794 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561f520f210 con 0x5561f559b480 2024-03-28T18:44:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.667+0000 7f18eeffd640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 0x5561f559d910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbff0 tx=0x7f18e00d6670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65f5e260cbec8239 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.667+0000 7f18ecff9640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 0x7f18d80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.667+0000 7f18effff640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 0x7f18d80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.668+0000 7f18d5ffb640 1 -- 172.21.15.145:0/3715426794 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00dd070 con 0x5561f559b480 2024-03-28T18:44:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.669+0000 7f18d5ffb640 1 -- 172.21.15.145:0/3715426794 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18e00d4e40 con 0x5561f559b480 2024-03-28T18:44:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.669+0000 7f18ecff9640 1 -- 172.21.15.145:0/1261181240 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f18e00ce960 con 0x7f18f010b010 2024-03-28T18:44:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.670+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3715426794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 msgr2=0x5561f559d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.670+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 0x5561f559d910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbff0 tx=0x7f18e00d6670 comp rx=0 tx=0).stop 2024-03-28T18:44:24.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.671+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18effff640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 0x7f18d80a4f50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f18e8036040 tx=0x7f18e80333e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3715426794 shutdown_connections 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f559b480 0x5561f559d910 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f18e00cbff0 tx=0x7f18e00d6670 comp rx=0 tx=0).stop 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f559ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3715426794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f5595d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.672+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3715426794 >> 172.21.15.145:0/3715426794 conn(0x5561f5590e90 msgr2=0x5561f5591290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.678+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3715426794 shutdown_connections 2024-03-28T18:44:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.683+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3715426794 wait complete. 2024-03-28T18:44:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.683+0000 7f18f8bc1740 1 -- start start 2024-03-28T18:44:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.683+0000 7f18f8bc1740 1 Processor -- start 2024-03-28T18:44:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5595920 0x5561f559fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5596310 0x5561f55a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f559e480 con 0x5561f5596310 2024-03-28T18:44:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f54b6d50 con 0x5561f55a5320 2024-03-28T18:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561f559e600 con 0x5561f5595920 2024-03-28T18:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.688+0000 7f18f8bc1740 10 client.0 _ll_register_callbacks cb 0x5561f54b6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:55066/0 (socket says 172.21.15.145:55066) 2024-03-28T18:44:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 -- 172.21.15.145:0/3613665643 learned_addr learned my addr 172.21.15.145:0/3613665643 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 -- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5595920 msgr2=0x5561f559fdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5595920 0x5561f559fdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 -- 172.21.15.145:0/3613665643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5596310 msgr2=0x5561f55a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5596310 0x5561f55a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.689+0000 7f18effff640 1 -- 172.21.15.145:0/3613665643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561f52db620 con 0x5561f55a5320 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18effff640 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f18e8035000 tx=0x7f18e802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61a2ce7488b6f792 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8051020 con 0x5561f55a5320 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561f5508050 con 0x5561f55a5320 2024-03-28T18:44:24.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f18e8003e00 con 0x5561f55a5320 2024-03-28T18:44:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e805c050 con 0x5561f55a5320 2024-03-28T18:44:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18f8bc1740 10 client.25622 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.690+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5561f55aab90 con 0x5561f55a5320 2024-03-28T18:44:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.691+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5561f5508050 con 0x5561f55a5320 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.692+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f18e8034870 con 0x5561f55a5320 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 10 client.25622 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 10 client.25622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 20 client.25622 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f55ac2c0 0x5561f55ae760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 4 client.25622 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.694+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561f55023a0 con 0x5561f55ac2c0 2024-03-28T18:44:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.697+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f55ac2c0 0x5561f55ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.708+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f55ac2c0 0x5561f55ae760 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=672c8dd61f7733ef server_cookie=43c6b78cc75f403b in_seq=0 out_seq=0 2024-03-28T18:44:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.708+0000 7f18ceffd640 10 client.25622 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.711+0000 7f18ceffd640 1 -- 172.21.15.145:0/3613665643 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5561f55023a0 con 0x5561f55ac2c0 2024-03-28T18:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.711+0000 7f18ceffd640 10 client.25622 handle_command_reply: tid=0 2024-03-28T18:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.711+0000 7f18f8bc1740 1 client.25622 shutdown 2024-03-28T18:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.711+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 msgr2=0x5561f55a7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.711+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f18e8035000 tx=0x7f18e802bf30 comp rx=0 tx=0).stop 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 shutdown_connections 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f55ac2c0 0x5561f55ae760 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f55a5320 0x5561f55a7710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5596310 0x5561f55a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/3613665643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5595920 0x5561f559fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 >> 172.21.15.145:0/3613665643 conn(0x5561f5590e90 msgr2=0x5561f5590020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.712+0000 7f18ceffd640 0 client.25622 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.713+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 shutdown_connections 2024-03-28T18:44:24.716 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.716+0000 7f18f8bc1740 1 -- 172.21.15.145:0/3613665643 wait complete. 2024-03-28T18:44:24.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.717+0000 7f18f8bc1740 20 client.25622 trim_cache size 0 max 16384 2024-03-28T18:44:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.761+0000 7f18f8bc1740 1 Processor -- start 2024-03-28T18:44:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 -- start start 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5595920 0x5561f5591270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f55a5320 0x5561f5597c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc00fa60 con 0x5561f55a5320 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0113e0 con 0x5561f5595920 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0100c0 con 0x5561f5596310 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18eeffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:49814/0 (socket says 172.21.15.145:49814) 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.769+0000 7f18eeffd640 1 -- 172.21.15.145:0/370394497 learned_addr learned my addr 172.21.15.145:0/370394497 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5595920 0x5561f5591270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 -- 172.21.15.145:0/370394497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5595920 msgr2=0x5561f5591270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5595920 0x5561f5591270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 -- 172.21.15.145:0/370394497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f55a5320 msgr2=0x5561f5597c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:44:24.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f55a5320 0x5561f5597c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.772 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 -- 172.21.15.145:0/370394497 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561f55a8b90 con 0x5561f5596310 2024-03-28T18:44:24.772 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.770+0000 7f18eeffd640 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009610 tx=0x7f18e00d4380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83866aa64d701870 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.774+0000 7f18d57fa640 1 -- 172.21.15.145:0/370394497 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0021070 con 0x5561f5596310 2024-03-28T18:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.774+0000 7f18d57fa640 1 -- 172.21.15.145:0/370394497 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18e00d6e70 con 0x5561f5596310 2024-03-28T18:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.776+0000 7f18f8bc1740 1 -- 172.21.15.145:0/370394497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 msgr2=0x5561f55976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.776+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009610 tx=0x7f18e00d4380 comp rx=0 tx=0).stop 2024-03-28T18:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.785+0000 7f18f8bc1740 1 -- 172.21.15.145:0/370394497 shutdown_connections 2024-03-28T18:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.785+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f55a5320 0x5561f5597c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.785+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f5596310 0x5561f55976d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.785+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/370394497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5595920 0x5561f5591270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.785+0000 7f18f8bc1740 1 -- 172.21.15.145:0/370394497 >> 172.21.15.145:0/370394497 conn(0x5561f5590e90 msgr2=0x5561f558beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.793+0000 7f18f8bc1740 1 -- 172.21.15.145:0/370394497 shutdown_connections 2024-03-28T18:44:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.797+0000 7f18f8bc1740 1 -- 172.21.15.145:0/370394497 wait complete. 2024-03-28T18:44:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.798+0000 7f18f8bc1740 1 -- start start 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.798+0000 7f18f8bc1740 1 Processor -- start 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f55bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5596310 0x5561f55bc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 0x5561f55bd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc010d80 con 0x5561f5595920 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc010720 con 0x5561f5596310 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc00dc20 con 0x5561f55a5320 2024-03-28T18:44:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18f8bc1740 10 client.0 _ll_register_callbacks cb 0x7f18e8033950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18effff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 0x5561f55bd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18effff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 0x5561f55bd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:49820/0 (socket says 172.21.15.145:49820) 2024-03-28T18:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.799+0000 7f18effff640 1 -- 172.21.15.145:0/954565875 learned_addr learned my addr 172.21.15.145:0/954565875 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.801+0000 7f18eeffd640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5596310 0x5561f55bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.801+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f55bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.803+0000 7f18ef7fe640 1 -- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 msgr2=0x5561f55bd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.803+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 0x5561f55bd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.803+0000 7f18ef7fe640 1 -- 172.21.15.145:0/954565875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5596310 msgr2=0x5561f55bc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.803+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5596310 0x5561f55bc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.803+0000 7f18ef7fe640 1 -- 172.21.15.145:0/954565875 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561f55bdc90 con 0x5561f5595920 2024-03-28T18:44:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.809+0000 7f18ef7fe640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f55bc240 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5561f55aefe0 tx=0x7f18e4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a343739635a7e004 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.809+0000 7f18ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.812+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4007d50 con 0x5561f5595920 2024-03-28T18:44:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.813+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f18e4019880 con 0x5561f5595920 2024-03-28T18:44:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.814+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18e00d6e70 con 0x5561f5595920 2024-03-28T18:44:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.818+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e4019a80 con 0x5561f5595920 2024-03-28T18:44:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.821+0000 7f18f8bc1740 10 client.15840 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.821+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5561f56072d0 con 0x5561f5595920 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.823+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18e4019c20 con 0x5561f5595920 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f18e4020020 con 0x5561f5595920 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 10 client.15840 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 20 client.15840 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f5607b60 0x5561f5609f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 4 client.15840 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:44:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.824+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f18e00d6e70 con 0x5561f5607b60 2024-03-28T18:44:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.825+0000 7f18eeffd640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f5607b60 0x5561f5609f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.830+0000 7f18eeffd640 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f5607b60 0x5561f5609f80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f265771dfd6679f server_cookie=98827899596fd0da in_seq=0 out_seq=0 2024-03-28T18:44:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:24.834+0000 7f18ceffd640 10 client.15840 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:25.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:24 smithi164 ceph-mon[36248]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 680 MiB/s wr, 1.29k op/s 2024-03-28T18:44:25.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:24 smithi145 ceph-mon[32151]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 680 MiB/s wr, 1.29k op/s 2024-03-28T18:44:25.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:24 smithi192 ceph-mon[36589]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 680 MiB/s wr, 1.29k op/s 2024-03-28T18:44:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.742+0000 7f18ceffd640 1 -- 172.21.15.145:0/954565875 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f18e00d6e70 con 0x5561f5607b60 2024-03-28T18:44:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.742+0000 7f18ceffd640 10 client.15840 handle_command_reply: tid=0 2024-03-28T18:44:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.742+0000 7f18f8bc1740 1 client.15840 shutdown 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 msgr2=0x5561f55bc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f55bc240 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5561f55aefe0 tx=0x7f18e4019240 comp rx=0 tx=0).stop 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 shutdown_connections 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5561f5607b60 0x5561f5609f80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5561f55a5320 0x5561f55bd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5561f5596310 0x5561f55bc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 --2- 172.21.15.145:0/954565875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5561f5595920 0x5561f55bc240 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18ceffd640 0 client.15840 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 >> 172.21.15.145:0/954565875 conn(0x5561f5590e90 msgr2=0x5561f5591270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 shutdown_connections 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 1 -- 172.21.15.145:0/954565875 wait complete. 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.743+0000 7f18f8bc1740 20 client.15840 trim_cache size 0 max 16384 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 msgr2=0x7f18d80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 0x7f18d80a4f50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f18e8036040 tx=0x7f18e80333e0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 msgr2=0x7f18f0115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 0x7f18f0115210 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f18e000d6d0 tx=0x7f18e000db90 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 shutdown_connections 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f010d9c0 0x7f18f01b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f18d80a2a90 0x7f18d80a4f50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f18f010b010 0x7f18f0115210 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 --2- 172.21.15.145:0/1261181240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f01086a0 0x7f18f0114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.745+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 >> 172.21.15.145:0/1261181240 conn(0x7f18f006e6e0 msgr2=0x7f18f0109490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.747+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 shutdown_connections 2024-03-28T18:44:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:26.747+0000 7f18f5fd4640 1 -- 172.21.15.145:0/1261181240 wait complete. 2024-03-28T18:44:27.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:27 smithi145 ceph-mon[32151]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 799 MiB/s wr, 1.53k op/s 2024-03-28T18:44:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:27 smithi192 ceph-mon[36589]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 799 MiB/s wr, 1.53k op/s 2024-03-28T18:44:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:27 smithi164 ceph-mon[36248]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 799 MiB/s wr, 1.53k op/s 2024-03-28T18:44:29.283 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:29 smithi145 ceph-mon[32151]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 734 MiB/s wr, 1.40k op/s 2024-03-28T18:44:29.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:29 smithi192 ceph-mon[36589]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 734 MiB/s wr, 1.40k op/s 2024-03-28T18:44:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:29 smithi164 ceph-mon[36248]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 734 MiB/s wr, 1.40k op/s 2024-03-28T18:44:31.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:31 smithi145 ceph-mon[32151]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 760 MiB/s wr, 1.42k op/s 2024-03-28T18:44:31.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:31 smithi192 ceph-mon[36589]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 760 MiB/s wr, 1.42k op/s 2024-03-28T18:44:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:31 smithi164 ceph-mon[36248]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 760 MiB/s wr, 1.42k op/s 2024-03-28T18:44:33.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:33 smithi192 ceph-mon[36589]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.32k op/s 2024-03-28T18:44:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:33 smithi164 ceph-mon[36248]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.32k op/s 2024-03-28T18:44:33.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:33 smithi145 ceph-mon[32151]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.32k op/s 2024-03-28T18:44:35.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:35 smithi192 ceph-mon[36589]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 651 MiB/s wr, 1.19k op/s 2024-03-28T18:44:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:35 smithi164 ceph-mon[36248]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 651 MiB/s wr, 1.19k op/s 2024-03-28T18:44:35.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:35 smithi145 ceph-mon[32151]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 651 MiB/s wr, 1.19k op/s 2024-03-28T18:44:37.333 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:37 smithi145 ceph-mon[32151]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 766 MiB/s wr, 1.41k op/s 2024-03-28T18:44:37.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:37 smithi192 ceph-mon[36589]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 766 MiB/s wr, 1.41k op/s 2024-03-28T18:44:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:37 smithi164 ceph-mon[36248]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 766 MiB/s wr, 1.41k op/s 2024-03-28T18:44:39.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:39 smithi192 ceph-mon[36589]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 712 MiB/s wr, 1.30k op/s 2024-03-28T18:44:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:39 smithi164 ceph-mon[36248]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 712 MiB/s wr, 1.30k op/s 2024-03-28T18:44:39.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:39 smithi145 ceph-mon[32151]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 712 MiB/s wr, 1.30k op/s 2024-03-28T18:44:39.936 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1061674477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 msgr2=0x7f02f4108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1061674477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f4108be0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009860 tx=0x7f02ec02bb70 comp rx=0 tx=0).stop 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1061674477 shutdown_connections 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1061674477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f4108be0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1061674477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f4109e10 0x7f02f41086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.936+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1061674477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4072b80 0x7f02f41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.937+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1061674477 >> 172.21.15.145:0/1061674477 conn(0x7f02f406e510 msgr2=0x7f02f4070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.937+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1061674477 shutdown_connections 2024-03-28T18:44:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.938+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1061674477 wait complete. 2024-03-28T18:44:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 Processor -- start 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 -- start start 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f4072b80 0x7f02f4110320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4109e10 0x7f02f4110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f410d750 con 0x7f02f4109e10 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f410d5d0 con 0x7f02f4072b80 2024-03-28T18:44:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.939+0000 7f02fa1e9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f410d8d0 con 0x7f02f410a290 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:33596/0 (socket says 172.21.15.145:33596) 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 -- 172.21.15.145:0/1559495037 learned_addr learned my addr 172.21.15.145:0/1559495037 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 -- 172.21.15.145:0/1559495037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f4072b80 msgr2=0x7f02f4110320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f4072b80 0x7f02f4110320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 -- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4109e10 msgr2=0x7f02f4110860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4109e10 0x7f02f4110860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.940+0000 7f02f99e8640 1 -- 172.21.15.145:0/1559495037 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f41b6a50 con 0x7f02f410a290 2024-03-28T18:44:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.941+0000 7f02f99e8640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02ec034c20 tx=0x7f02ec02c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.941+0000 7f02ea7fc640 1 -- 172.21.15.145:0/1559495037 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec039040 con 0x7f02f410a290 2024-03-28T18:44:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.942+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f41b6c20 con 0x7f02f410a290 2024-03-28T18:44:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.942+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f41b70e0 con 0x7f02f410a290 2024-03-28T18:44:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.942+0000 7f02ea7fc640 1 -- 172.21.15.145:0/1559495037 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02ec042070 con 0x7f02f410a290 2024-03-28T18:44:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.943+0000 7f02ea7fc640 1 -- 172.21.15.145:0/1559495037 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0039f0 con 0x7f02f410a290 2024-03-28T18:44:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.945+0000 7f02ea7fc640 1 -- 172.21.15.145:0/1559495037 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f02ec003c50 con 0x7f02f410a290 2024-03-28T18:44:39.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.947+0000 7f02ea7fc640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 0x7f02e00a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.947 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.947+0000 7f02ea7fc640 1 -- 172.21.15.145:0/1559495037 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f02ec0f0840 con 0x7f02f410a290 2024-03-28T18:44:39.949 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.949+0000 7f02f91e7640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 0x7f02e00a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.949 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.950+0000 7f02f91e7640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 0x7f02e00a4cb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f02f0005e30 tx=0x7f02f0005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.952+0000 7f02fe03b740 1 Processor -- start 2024-03-28T18:44:39.953 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.952+0000 7f02fe03b740 1 -- start start 2024-03-28T18:44:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.953+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d660af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.957+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6610010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.957+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.957+0000 7f02fe03b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d656fb10 con 0x5633d660b510 2024-03-28T18:44:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.957+0000 7f02fe03b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d6518900 con 0x5633d660ab20 2024-03-28T18:44:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.957+0000 7f02fe03b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d656bbb0 con 0x5633d6610680 2024-03-28T18:44:39.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:33608/0 (socket says 172.21.15.145:33608) 2024-03-28T18:44:39.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 -- 172.21.15.145:0/2484354373 learned_addr learned my addr 172.21.15.145:0/2484354373 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:39.963 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f89e6640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6610010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f91e7640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d660af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 -- 172.21.15.145:0/2484354373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 msgr2=0x5633d660af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d660af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 -- 172.21.15.145:0/2484354373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 msgr2=0x5633d6610010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6610010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 -- 172.21.15.145:0/2484354373 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633d6284210 con 0x5633d6610680 2024-03-28T18:44:39.964 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f89e6640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6610010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02f99e8640 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0c50a0 tx=0x7f02ec0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404a5bcea27a7dbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.960+0000 7f02df7fe640 1 -- 172.21.15.145:0/2484354373 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec032340 con 0x5633d6610680 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.961+0000 7f02df7fe640 1 -- 172.21.15.145:0/2484354373 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02ec034e50 con 0x5633d6610680 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.961+0000 7f02df7fe640 1 -- 172.21.15.145:0/2484354373 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec02c940 con 0x5633d6610680 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 -- 172.21.15.145:0/2484354373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 msgr2=0x5633d6612b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:39.965 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0c50a0 tx=0x7f02ec0095b0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 -- 172.21.15.145:0/2484354373 shutdown_connections 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d6612b10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6610010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 --2- 172.21.15.145:0/2484354373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d660af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.962+0000 7f02fe03b740 1 -- 172.21.15.145:0/2484354373 >> 172.21.15.145:0/2484354373 conn(0x5633d6606090 msgr2=0x5633d6606490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:39.966 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.963+0000 7f02fe03b740 1 -- 172.21.15.145:0/2484354373 shutdown_connections 2024-03-28T18:44:39.968 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.964+0000 7f02fe03b740 1 -- 172.21.15.145:0/2484354373 wait complete. 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.964+0000 7f02fe03b740 1 -- start start 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.964+0000 7f02fe03b740 1 Processor -- start 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d6614ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6615020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d661c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d66133f0 con 0x5633d660b510 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d6614ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d6614ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57412/0 (socket says 172.21.15.145:57412) 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02f91e7640 1 -- 172.21.15.145:0/4179205294 learned_addr learned my addr 172.21.15.145:0/4179205294 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02f99e8640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d661c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d652bd50 con 0x5633d660ab20 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633d6613570 con 0x5633d6610680 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.965+0000 7f02fe03b740 10 client.0 _ll_register_callbacks cb 0x5633d652b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.966+0000 7f02f99e8640 1 -- 172.21.15.145:0/4179205294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 msgr2=0x5633d6614ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.966+0000 7f02f99e8640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d6614ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.966+0000 7f02f99e8640 1 -- 172.21.15.145:0/4179205294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 msgr2=0x5633d6615020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:44:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.968+0000 7f02f99e8640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6615020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.968+0000 7f02f99e8640 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633d6350620 con 0x5633d6610680 2024-03-28T18:44:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.968+0000 7f02f99e8640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d661c060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0edfe0 tx=0x7f02ec0323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f3dfd54a08fd666 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:39.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.969+0000 7f02dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:44:39.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.970+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0db050 con 0x5633d6610680 2024-03-28T18:44:39.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.970+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f02ec0bb130 con 0x5633d6610680 2024-03-28T18:44:39.972 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.972+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02ec034e50 con 0x5633d6610680 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02ec0bd530 con 0x5633d6610680 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 10 client.25720 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5633d66133f0 con 0x5633d6610680 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02ec044310 con 0x5633d6610680 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f02ec03c070 con 0x5633d6610680 2024-03-28T18:44:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 10 client.25720 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 10 client.25720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 20 client.25720 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.973+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d6620890 0x5633d6622cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.974+0000 7f02fe03b740 4 client.25720 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.974+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02ec034e50 con 0x5633d6620890 2024-03-28T18:44:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.975+0000 7f02f91e7640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d6620890 0x5633d6622cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.976+0000 7f02dcff9640 10 client.25720 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.976+0000 7f02f91e7640 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d6620890 0x5633d6622cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94fcc168680cd35 server_cookie=7b196c0217a046bd in_seq=0 out_seq=0 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.977+0000 7f02dcff9640 1 -- 172.21.15.145:0/4179205294 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f02ec034e50 con 0x5633d6620890 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.977+0000 7f02dcff9640 10 client.25720 handle_command_reply: tid=0 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.977+0000 7f02fe03b740 1 client.25720 shutdown 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.977+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 msgr2=0x5633d661c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.977+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d661c060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0edfe0 tx=0x7f02ec0323c0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 shutdown_connections 2024-03-28T18:44:39.980 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d6620890 0x5633d6622cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02dcff9640 0 client.25720 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d6610680 0x5633d661c060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660b510 0x5633d6615020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 --2- 172.21.15.145:0/4179205294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660ab20 0x5633d6614ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.978+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 >> 172.21.15.145:0/4179205294 conn(0x5633d6606090 msgr2=0x5633d6605290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.979+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 shutdown_connections 2024-03-28T18:44:39.981 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.979+0000 7f02fe03b740 1 -- 172.21.15.145:0/4179205294 wait complete. 2024-03-28T18:44:39.982 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:39.981+0000 7f02fe03b740 20 client.25720 trim_cache size 0 max 16384 2024-03-28T18:44:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 Processor -- start 2024-03-28T18:44:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 -- start start 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660b510 0x5633d661c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d6610680 0x5633d6627590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:33614/0 (socket says 172.21.15.145:33614) 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02f91e7640 1 -- 172.21.15.145:0/3489220049 learned_addr learned my addr 172.21.15.145:0/3489220049 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d800fac0 con 0x5633d6610680 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8011440 con 0x5633d660b510 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.006+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8010120 con 0x5633d660ab20 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.007+0000 7f02f91e7640 1 -- 172.21.15.145:0/3489220049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660b510 msgr2=0x5633d661c020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.007+0000 7f02f91e7640 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660b510 0x5633d661c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.007+0000 7f02f91e7640 1 -- 172.21.15.145:0/3489220049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d6610680 msgr2=0x5633d6627590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.007+0000 7f02f91e7640 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d6610680 0x5633d6627590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.007+0000 7f02f91e7640 1 -- 172.21.15.145:0/3489220049 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633d6623310 con 0x5633d660ab20 2024-03-28T18:44:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.008+0000 7f02f91e7640 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5633d6623510 tx=0x7f02f0011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c3e9469b00b9074 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.008+0000 7f02deffd640 1 -- 172.21.15.145:0/3489220049 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f002a070 con 0x5633d660ab20 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.008+0000 7f02deffd640 1 -- 172.21.15.145:0/3489220049 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f0025a00 con 0x5633d660ab20 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.008+0000 7f02deffd640 1 -- 172.21.15.145:0/3489220049 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f001c850 con 0x5633d660ab20 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.010+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 msgr2=0x5633d66144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.010+0000 7f02fe03b740 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5633d6623510 tx=0x7f02f0011000 comp rx=0 tx=0).stop 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 shutdown_connections 2024-03-28T18:44:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d6610680 0x5633d6627590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d660b510 0x5633d661c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 --2- 172.21.15.145:0/3489220049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660ab20 0x5633d66144c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 >> 172.21.15.145:0/3489220049 conn(0x5633d6606090 msgr2=0x5633d6613db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 shutdown_connections 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 -- 172.21.15.145:0/3489220049 wait complete. 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 -- start start 2024-03-28T18:44:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.011+0000 7f02fe03b740 1 Processor -- start 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660b510 0x5633d661a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d6610680 0x5633d661b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8010de0 con 0x5633d660ab20 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8010780 con 0x5633d6610680 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d800dc80 con 0x5633d660b510 2024-03-28T18:44:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02fe03b740 10 client.0 _ll_register_callbacks cb 0x7f02ec0c5640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02f91e7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45000/0 (socket says 172.21.15.145:45000) 2024-03-28T18:44:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.012+0000 7f02f91e7640 1 -- 172.21.15.145:0/96158670 learned_addr learned my addr 172.21.15.145:0/96158670 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f99e8640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d6610680 0x5633d661b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f89e6640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660b510 0x5633d661a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f91e7640 1 -- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660b510 msgr2=0x5633d661a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f91e7640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660b510 0x5633d661a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f91e7640 1 -- 172.21.15.145:0/96158670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d6610680 msgr2=0x5633d661b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f91e7640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d6610680 0x5633d661b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.013+0000 7f02f91e7640 1 -- 172.21.15.145:0/96158670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633d661b660 con 0x5633d660ab20 2024-03-28T18:44:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.014+0000 7f02f91e7640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f02f0000f80 tx=0x7f02f0025760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21042fc4afd5654 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.014+0000 7f02dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.014+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f002a070 con 0x5633d660ab20 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.014+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f02f000f610 con 0x5633d660ab20 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.014+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02f0025a00 con 0x5633d660ab20 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.015+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02f000f890 con 0x5633d660ab20 2024-03-28T18:44:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.016+0000 7f02fe03b740 10 client.15846 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.016+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5633d667c480 con 0x5633d660ab20 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f0025a00 con 0x5633d660ab20 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f02f0030b00 con 0x5633d660ab20 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 10 client.15846 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 20 client.15846 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d667cdb0 0x5633d667f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 4 client.15846 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:44:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5633d657d050 con 0x5633d667cdb0 2024-03-28T18:44:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.017+0000 7f02f89e6640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d667cdb0 0x5633d667f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.018+0000 7f02dcff9640 10 client.15846 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:40.018+0000 7f02f89e6640 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d667cdb0 0x5633d667f1d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b2162d52d614b2 server_cookie=61b6a1d1b39aa788 in_seq=0 out_seq=0 2024-03-28T18:44:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:41 smithi192 ceph-mon[36589]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 706 MiB/s wr, 1.40k op/s 2024-03-28T18:44:41.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:41 smithi164 ceph-mon[36248]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 706 MiB/s wr, 1.40k op/s 2024-03-28T18:44:41.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:41 smithi145 ceph-mon[32151]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 706 MiB/s wr, 1.40k op/s 2024-03-28T18:44:42.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.810+0000 7f02dcff9640 1 -- 172.21.15.145:0/96158670 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5633d657d050 con 0x5633d667cdb0 2024-03-28T18:44:42.811 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.810+0000 7f02dcff9640 10 client.15846 handle_command_reply: tid=0 2024-03-28T18:44:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.810+0000 7f02fe03b740 1 client.15846 shutdown 2024-03-28T18:44:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 msgr2=0x5633d6619b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:42.812 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f02f0000f80 tx=0x7f02f0025760 comp rx=0 tx=0).stop 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 shutdown_connections 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5633d667cdb0 0x5633d667f1d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5633d6610680 0x5633d661b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5633d660b510 0x5633d661a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 --2- 172.21.15.145:0/96158670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5633d660ab20 0x5633d6619b20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 >> 172.21.15.145:0/96158670 conn(0x5633d6606090 msgr2=0x5633d6604a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02dcff9640 0 client.15846 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 shutdown_connections 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.811+0000 7f02fe03b740 1 -- 172.21.15.145:0/96158670 wait complete. 2024-03-28T18:44:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.812+0000 7f02fe03b740 20 client.15846 trim_cache size 0 max 16384 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 msgr2=0x7f02e00a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 0x7f02e00a4cb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f02f0005e30 tx=0x7f02f0005d80 comp rx=0 tx=0).stop 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 msgr2=0x7f02f41178d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02ec034c20 tx=0x7f02ec02c510 comp rx=0 tx=0).stop 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 shutdown_connections 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02e00a27f0 0x7f02e00a4cb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02f410a290 0x7f02f41178d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4109e10 0x7f02f4110860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 --2- 172.21.15.145:0/1559495037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f4072b80 0x7f02f4110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:42.815 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.813+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 >> 172.21.15.145:0/1559495037 conn(0x7f02f406e510 msgr2=0x7f02f4070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.816+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 shutdown_connections 2024-03-28T18:44:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:42.816+0000 7f02fa1e9640 1 -- 172.21.15.145:0/1559495037 wait complete. 2024-03-28T18:44:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:43 smithi192 ceph-mon[36589]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 679 MiB/s wr, 1.50k op/s 2024-03-28T18:44:43.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:43 smithi164 ceph-mon[36248]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 679 MiB/s wr, 1.50k op/s 2024-03-28T18:44:43.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:43 smithi145 ceph-mon[32151]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 679 MiB/s wr, 1.50k op/s 2024-03-28T18:44:45.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:45 smithi192 ceph-mon[36589]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 631 MiB/s wr, 1.49k op/s 2024-03-28T18:44:45.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:45 smithi164 ceph-mon[36248]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 631 MiB/s wr, 1.49k op/s 2024-03-28T18:44:45.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:45 smithi145 ceph-mon[32151]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 631 MiB/s wr, 1.49k op/s 2024-03-28T18:44:47.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:47 smithi192 ceph-mon[36589]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 705 MiB/s wr, 1.53k op/s 2024-03-28T18:44:47.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:47 smithi164 ceph-mon[36248]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 705 MiB/s wr, 1.53k op/s 2024-03-28T18:44:47.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:47 smithi145 ceph-mon[32151]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 705 MiB/s wr, 1.53k op/s 2024-03-28T18:44:48.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:48 smithi192 ceph-mon[36589]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.29k op/s 2024-03-28T18:44:48.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:48 smithi164 ceph-mon[36248]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.29k op/s 2024-03-28T18:44:48.537 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:48 smithi145 ceph-mon[32151]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.29k op/s 2024-03-28T18:44:49.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:49 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:49 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:49.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:49 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:50.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:50 smithi192 ceph-mon[36589]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 675 MiB/s wr, 1.29k op/s 2024-03-28T18:44:50.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:50 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:50 smithi164 ceph-mon[36248]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 675 MiB/s wr, 1.29k op/s 2024-03-28T18:44:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:50 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:50 smithi145 ceph-mon[32151]: pgmap v563: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 675 MiB/s wr, 1.29k op/s 2024-03-28T18:44:50.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:50.779 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:50 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi164 to 174.1M 2024-03-28T18:44:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: Unable to set osd_memory_target on smithi164 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-28T18:44:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:51.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:51 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi164 to 174.1M 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: Unable to set osd_memory_target on smithi164 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:51.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:51 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:44:51.831 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi164 to 174.1M 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: Unable to set osd_memory_target on smithi164 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:51.832 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:51 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:44:52.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:52 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi145 to 176.2M 2024-03-28T18:44:52.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:52 smithi192 ceph-mon[36589]: Unable to set osd_memory_target on smithi145 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-03-28T18:44:52.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:52 smithi192 ceph-mon[36589]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 675 MiB/s wr, 1.25k op/s 2024-03-28T18:44:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:52 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi145 to 176.2M 2024-03-28T18:44:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:52 smithi164 ceph-mon[36248]: Unable to set osd_memory_target on smithi145 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-03-28T18:44:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:52 smithi164 ceph-mon[36248]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 675 MiB/s wr, 1.25k op/s 2024-03-28T18:44:52.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:52 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi145 to 176.2M 2024-03-28T18:44:52.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:52 smithi145 ceph-mon[32151]: Unable to set osd_memory_target on smithi145 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-03-28T18:44:52.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:52 smithi145 ceph-mon[32151]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 675 MiB/s wr, 1.25k op/s 2024-03-28T18:44:55.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:54 smithi192 ceph-mon[36589]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 644 MiB/s wr, 1.13k op/s 2024-03-28T18:44:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:54 smithi164 ceph-mon[36248]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 644 MiB/s wr, 1.13k op/s 2024-03-28T18:44:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:54 smithi145 ceph-mon[32151]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 644 MiB/s wr, 1.13k op/s 2024-03-28T18:44:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.069+0000 7f679beb2640 1 -- 172.21.15.145:0/3790865723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 msgr2=0x7f679410ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.069+0000 7f679beb2640 1 --2- 172.21.15.145:0/3790865723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679410ad00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6790009370 tx=0x7f679002cea0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.074+0000 7f679beb2640 1 -- 172.21.15.145:0/3790865723 shutdown_connections 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.074+0000 7f679beb2640 1 --2- 172.21.15.145:0/3790865723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 0x7f6794110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.074+0000 7f679beb2640 1 --2- 172.21.15.145:0/3790865723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 0x7f679410d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.074+0000 7f679beb2640 1 --2- 172.21.15.145:0/3790865723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679410ad00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.074+0000 7f679beb2640 1 -- 172.21.15.145:0/3790865723 >> 172.21.15.145:0/3790865723 conn(0x7f679406e8a0 msgr2=0x7f6794070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.075+0000 7f679beb2640 1 -- 172.21.15.145:0/3790865723 shutdown_connections 2024-03-28T18:44:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.078+0000 7f679beb2640 1 -- 172.21.15.145:0/3790865723 wait complete. 2024-03-28T18:44:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.079+0000 7f679beb2640 1 Processor -- start 2024-03-28T18:44:56.079 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 -- start start 2024-03-28T18:44:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 0x7f679419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 0x7f67941a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794113aa0 con 0x7f6794108910 2024-03-28T18:44:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794113920 con 0x7f679410dd70 2024-03-28T18:44:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.080+0000 7f679beb2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794113c20 con 0x7f679410b2d0 2024-03-28T18:44:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.082+0000 7f679a44f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 0x7f67941a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.082+0000 7f679944d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 0x7f679419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.082+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.082+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:57486/0 (socket says 172.21.15.145:57486) 2024-03-28T18:44:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.082+0000 7f6799c4e640 1 -- 172.21.15.145:0/1099783447 learned_addr learned my addr 172.21.15.145:0/1099783447 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 -- 172.21.15.145:0/1099783447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 msgr2=0x7f679419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 0x7f679419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 -- 172.21.15.145:0/1099783447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 msgr2=0x7f67941a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 0x7f67941a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 -- 172.21.15.145:0/1099783447 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67941a5cd0 con 0x7f6794108910 2024-03-28T18:44:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.083+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6790005440 tx=0x7f6790036980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.088+0000 7f678affd640 1 -- 172.21.15.145:0/1099783447 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679003a040 con 0x7f6794108910 2024-03-28T18:44:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.088+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67941a5f60 con 0x7f6794108910 2024-03-28T18:44:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.088+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67941a63e0 con 0x7f6794108910 2024-03-28T18:44:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.088+0000 7f678affd640 1 -- 172.21.15.145:0/1099783447 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6790043030 con 0x7f6794108910 2024-03-28T18:44:56.090 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.088+0000 7f678affd640 1 -- 172.21.15.145:0/1099783447 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790034a10 con 0x7f6794108910 2024-03-28T18:44:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.096+0000 7f679eb7f740 1 Processor -- start 2024-03-28T18:44:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 -- start start 2024-03-28T18:44:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614039550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61403e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614041120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e613f95b10 con 0x55e614039130 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e613f3e900 con 0x55e614039b20 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.097+0000 7f679eb7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e613f91bb0 con 0x55e61403ec90 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.098+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61403e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.098+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61403e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57714/0 (socket says 172.21.15.145:57714) 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.098+0000 7f6799c4e640 1 -- 172.21.15.145:0/1240463270 learned_addr learned my addr 172.21.15.145:0/1240463270 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.098+0000 7f679944d640 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614041120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.098+0000 7f679a44f640 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614039550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.099+0000 7f679944d640 1 -- 172.21.15.145:0/1240463270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 msgr2=0x55e61403e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.099+0000 7f679944d640 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61403e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.099+0000 7f679944d640 1 -- 172.21.15.145:0/1240463270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 msgr2=0x55e614039550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.099+0000 7f679944d640 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614039550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.099+0000 7f679944d640 1 -- 172.21.15.145:0/1240463270 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e613caa210 con 0x55e61403ec90 2024-03-28T18:44:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.102+0000 7f678affd640 1 -- 172.21.15.145:0/1099783447 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6790034bb0 con 0x7f6794108910 2024-03-28T18:44:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.103+0000 7f678affd640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 0x7f67800a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.103+0000 7f678affd640 1 -- 172.21.15.145:0/1099783447 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f67900f7920 con 0x7f6794108910 2024-03-28T18:44:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.103+0000 7f679944d640 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614041120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f679419f5c0 tx=0x7f678400acc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0bc8865104ee544 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.103+0000 7f679a44f640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 0x7f67800a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.104+0000 7f679a44f640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 0x7f67800a4fa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55e61403a380 tx=0x7f678c00e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.107+0000 7f677ffff640 1 -- 172.21.15.145:0/1240463270 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784015070 con 0x55e61403ec90 2024-03-28T18:44:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.107+0000 7f677ffff640 1 -- 172.21.15.145:0/1240463270 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67840113f0 con 0x55e61403ec90 2024-03-28T18:44:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.107+0000 7f677ffff640 1 -- 172.21.15.145:0/1240463270 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67840185c0 con 0x55e61403ec90 2024-03-28T18:44:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 -- 172.21.15.145:0/1240463270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 msgr2=0x55e614041120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614041120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f679419f5c0 tx=0x7f678400acc0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 -- 172.21.15.145:0/1240463270 shutdown_connections 2024-03-28T18:44:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614041120 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61403e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1240463270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614039550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.110+0000 7f679eb7f740 1 -- 172.21.15.145:0/1240463270 >> 172.21.15.145:0/1240463270 conn(0x55e6140346a0 msgr2=0x55e614034aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.111+0000 7f679eb7f740 1 -- 172.21.15.145:0/1240463270 shutdown_connections 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.114+0000 7f679eb7f740 1 -- 172.21.15.145:0/1240463270 wait complete. 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.115+0000 7f679eb7f740 1 -- start start 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.115+0000 7f679eb7f740 1 Processor -- start 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e6140434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039b20 0x55e614043a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e614041c40 con 0x55e614039b20 2024-03-28T18:44:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e613f51d50 con 0x55e614039130 2024-03-28T18:44:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e614041dc0 con 0x55e61403ec90 2024-03-28T18:44:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679eb7f740 10 client.0 _ll_register_callbacks cb 0x55e613f51720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.116+0000 7f679a44f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:37304/0 (socket says 172.21.15.145:37304) 2024-03-28T18:44:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 -- 172.21.15.145:0/3146868125 learned_addr learned my addr 172.21.15.145:0/3146868125 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 -- 172.21.15.145:0/3146868125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 msgr2=0x55e6140434c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e6140434c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 -- 172.21.15.145:0/3146868125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039b20 msgr2=0x55e614043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039b20 0x55e614043a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.117+0000 7f679a44f640 1 -- 172.21.15.145:0/3146868125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e613d76620 con 0x55e61403ec90 2024-03-28T18:44:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.118+0000 7f679a44f640 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f678c00afd0 tx=0x7f678c004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0c1198087fa56c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.125+0000 7f677d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:44:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.125+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c016780 con 0x55e61403ec90 2024-03-28T18:44:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.125+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f678c016920 con 0x55e61403ec90 2024-03-28T18:44:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.127+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67840113f0 con 0x55e61403ec90 2024-03-28T18:44:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.128+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f678c016b20 con 0x55e61403ec90 2024-03-28T18:44:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.131+0000 7f679eb7f740 10 client.25732 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.131+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e61404dde0 con 0x55e61403ec90 2024-03-28T18:44:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.133+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67840113f0 con 0x55e61403ec90 2024-03-28T18:44:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.133+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f678c029350 con 0x55e61403ec90 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 10 client.25732 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 20 client.25732 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e61404f4d0 0x55e614051990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 4 client.25732 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:56.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.134+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e613fa3050 con 0x55e61404f4d0 2024-03-28T18:44:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.140+0000 7f6799c4e640 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e61404f4d0 0x55e614051990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.150+0000 7f677d7fa640 10 client.25732 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.150+0000 7f6799c4e640 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e61404f4d0 0x55e614051990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1be9ae66a5f7da4 server_cookie=c0eacd1bead7b2d2 in_seq=0 out_seq=0 2024-03-28T18:44:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.155+0000 7f677d7fa640 1 -- 172.21.15.145:0/3146868125 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e613fa3050 con 0x55e61404f4d0 2024-03-28T18:44:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.156+0000 7f677d7fa640 10 client.25732 handle_command_reply: tid=0 2024-03-28T18:44:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.156+0000 7f679eb7f740 1 client.25732 shutdown 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.167+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 msgr2=0x55e61404a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.167+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f678c00afd0 tx=0x7f678c004a80 comp rx=0 tx=0).stop 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 shutdown_connections 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e61404f4d0 0x55e614051990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e61404a9d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039b20 0x55e614043a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 --2- 172.21.15.145:0/3146868125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e6140434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f677d7fa640 0 client.25732 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:56.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.168+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 >> 172.21.15.145:0/3146868125 conn(0x55e6140346a0 msgr2=0x55e61402b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:56.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.170+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 shutdown_connections 2024-03-28T18:44:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.172+0000 7f679eb7f740 1 -- 172.21.15.145:0/3146868125 wait complete. 2024-03-28T18:44:56.174 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.172+0000 7f679eb7f740 20 client.25732 trim_cache size 0 max 16384 2024-03-28T18:44:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 Processor -- start 2024-03-28T18:44:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 -- start start 2024-03-28T18:44:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614037900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.218 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61404a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614037e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677800fb40 con 0x55e614039130 2024-03-28T18:44:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780114c0 con 0x55e614039b20 2024-03-28T18:44:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.218+0000 7f679eb7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67780101a0 con 0x55e61403ec90 2024-03-28T18:44:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614037900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614037900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:57518/0 (socket says 172.21.15.145:57518) 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f6799c4e640 1 -- 172.21.15.145:0/2114615242 learned_addr learned my addr 172.21.15.145:0/2114615242 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679944d640 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61404a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614037e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 -- 172.21.15.145:0/2114615242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 msgr2=0x55e61404a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61404a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 -- 172.21.15.145:0/2114615242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 msgr2=0x55e614037900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614037900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.219+0000 7f679a44f640 1 -- 172.21.15.145:0/2114615242 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e614039f00 con 0x55e61403ec90 2024-03-28T18:44:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.222+0000 7f679a44f640 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614037e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f678c0297f0 tx=0x7f678c029af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b339a2444cb999b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.223+0000 7f677f7fe640 1 -- 172.21.15.145:0/2114615242 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c029b60 con 0x55e61403ec90 2024-03-28T18:44:56.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.223+0000 7f677f7fe640 1 -- 172.21.15.145:0/2114615242 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f678c03b070 con 0x55e61403ec90 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 -- 172.21.15.145:0/2114615242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 msgr2=0x55e614037e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614037e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f678c0297f0 tx=0x7f678c029af0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 -- 172.21.15.145:0/2114615242 shutdown_connections 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e61403ec90 0x55e614037e40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039b20 0x55e61404a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 --2- 172.21.15.145:0/2114615242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e614039130 0x55e614037900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.224+0000 7f679eb7f740 1 -- 172.21.15.145:0/2114615242 >> 172.21.15.145:0/2114615242 conn(0x55e6140346a0 msgr2=0x55e614026eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.226+0000 7f679eb7f740 1 -- 172.21.15.145:0/2114615242 shutdown_connections 2024-03-28T18:44:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.226+0000 7f679eb7f740 1 -- 172.21.15.145:0/2114615242 wait complete. 2024-03-28T18:44:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.226+0000 7f679eb7f740 1 -- start start 2024-03-28T18:44:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 Processor -- start 2024-03-28T18:44:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e614039b20 0x55e6140473b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e6140478f0 0x55e61402e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778010e60 con 0x55e6140478f0 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778010800 con 0x55e614039130 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677800dd00 con 0x55e614039b20 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.227+0000 7f679eb7f740 10 client.0 _ll_register_callbacks cb 0x7f678c00ac80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.228+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.228+0000 7f6799c4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57734/0 (socket says 172.21.15.145:57734) 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.228+0000 7f6799c4e640 1 -- 172.21.15.145:0/1490106570 learned_addr learned my addr 172.21.15.145:0/1490106570 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:44:56.228 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.228+0000 7f679a44f640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e6140478f0 0x55e61402e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.230+0000 7f6799c4e640 1 -- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e614039b20 msgr2=0x55e6140473b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.230+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e614039b20 0x55e6140473b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.230+0000 7f6799c4e640 1 -- 172.21.15.145:0/1490106570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e6140478f0 msgr2=0x55e61402e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.230+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e6140478f0 0x55e61402e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.230+0000 7f6799c4e640 1 -- 172.21.15.145:0/1490106570 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e61402ecf0 con 0x55e614039130 2024-03-28T18:44:56.230 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.231+0000 7f6799c4e640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67900c3bb0 tx=0x7f6790005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fa8da6c6e9707c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:56.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.235+0000 7f677d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:44:56.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.235+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67900c3810 con 0x55e614039130 2024-03-28T18:44:56.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.236+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f67900f5120 con 0x55e614039130 2024-03-28T18:44:56.236 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.237+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e613f9d3a0 con 0x55e614039130 2024-03-28T18:44:56.237 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.238+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67900c1ff0 con 0x55e614039130 2024-03-28T18:44:56.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.240+0000 7f679eb7f740 10 client.25646 fetch_fsmap learned FSMap version 31 2024-03-28T18:44:56.240 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.240+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e614030160 con 0x55e614039130 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e613f9d3a0 con 0x55e614039130 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f67900c0490 con 0x55e614039130 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 10 client.25646 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 10 client.25646 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 20 client.25646 populate_metadata read hostname 'smithi145' 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e614030b40 0x55e614032f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 4 client.25646 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:44:56.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.243+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e613f1b710 con 0x55e614030b40 2024-03-28T18:44:56.246 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.246+0000 7f679944d640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e614030b40 0x55e614032f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.255+0000 7f677d7fa640 10 client.25646 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:56.255+0000 7f679944d640 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e614030b40 0x55e614032f60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e57f89311a1578b5 server_cookie=ecda249297c94eef in_seq=0 out_seq=0 2024-03-28T18:44:57.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:56 smithi192 ceph-mon[36589]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 723 MiB/s wr, 1.23k op/s 2024-03-28T18:44:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:56 smithi164 ceph-mon[36248]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 723 MiB/s wr, 1.23k op/s 2024-03-28T18:44:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:56 smithi145 ceph-mon[32151]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 723 MiB/s wr, 1.23k op/s 2024-03-28T18:44:59.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:44:58 smithi192 ceph-mon[36589]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 708 MiB/s wr, 1.35k op/s 2024-03-28T18:44:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:44:58 smithi164 ceph-mon[36248]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 708 MiB/s wr, 1.35k op/s 2024-03-28T18:44:59.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:44:58 smithi145 ceph-mon[32151]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 708 MiB/s wr, 1.35k op/s 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.303+0000 7f677d7fa640 1 -- 172.21.15.145:0/1490106570 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e613f1b710 con 0x55e614030b40 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.303+0000 7f677d7fa640 10 client.25646 handle_command_reply: tid=0 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.303+0000 7f679eb7f740 1 client.25646 shutdown 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 msgr2=0x55e61404a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f67900c3bb0 tx=0x7f6790005880 comp rx=0 tx=0).stop 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 shutdown_connections 2024-03-28T18:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55e614030b40 0x55e614032f60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e6140478f0 0x55e61402e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e614039b20 0x55e6140473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 --2- 172.21.15.145:0/1490106570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e614039130 0x55e61404a910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 >> 172.21.15.145:0/1490106570 conn(0x55e6140346a0 msgr2=0x55e614037900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f677d7fa640 0 client.25646 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 shutdown_connections 2024-03-28T18:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.304+0000 7f679eb7f740 1 -- 172.21.15.145:0/1490106570 wait complete. 2024-03-28T18:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.305+0000 7f679eb7f740 20 client.25646 trim_cache size 0 max 16384 2024-03-28T18:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:44:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.308+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 msgr2=0x7f67800a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.308+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 0x7f67800a4fa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55e61403a380 tx=0x7f678c00e040 comp rx=0 tx=0).stop 2024-03-28T18:44:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.308+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 msgr2=0x7f679419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.308+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6790005440 tx=0x7f6790036980 comp rx=0 tx=0).stop 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 shutdown_connections 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679410dd70 0x7f67941a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f67800a2ae0 0x7f67800a4fa0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679410b2d0 0x7f679419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 --2- 172.21.15.145:0/1099783447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6794108910 0x7f679419e100 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.310+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 >> 172.21.15.145:0/1099783447 conn(0x7f679406e8a0 msgr2=0x7f6794109250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.311+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 shutdown_connections 2024-03-28T18:44:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:44:59.311+0000 7f679beb2640 1 -- 172.21.15.145:0/1099783447 wait complete. 2024-03-28T18:45:01.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:00 smithi192 ceph-mon[36589]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 705 MiB/s wr, 1.56k op/s 2024-03-28T18:45:01.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:00 smithi164 ceph-mon[36248]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 705 MiB/s wr, 1.56k op/s 2024-03-28T18:45:01.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:00 smithi145 ceph-mon[32151]: pgmap v568: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 705 MiB/s wr, 1.56k op/s 2024-03-28T18:45:03.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:02 smithi164 ceph-mon[36248]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 660 MiB/s wr, 1.60k op/s 2024-03-28T18:45:03.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:02 smithi145 ceph-mon[32151]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 660 MiB/s wr, 1.60k op/s 2024-03-28T18:45:03.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:02 smithi192 ceph-mon[36589]: pgmap v569: 129 pgs: 129 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 660 MiB/s wr, 1.60k op/s 2024-03-28T18:45:05.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:04 smithi164 ceph-mon[36248]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 1.40k op/s 2024-03-28T18:45:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:04 smithi145 ceph-mon[32151]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 1.40k op/s 2024-03-28T18:45:05.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:04 smithi192 ceph-mon[36589]: pgmap v570: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 MiB/s wr, 1.40k op/s 2024-03-28T18:45:07.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:06 smithi192 ceph-mon[36589]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 720 MiB/s wr, 1.34k op/s 2024-03-28T18:45:07.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:06 smithi164 ceph-mon[36248]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 720 MiB/s wr, 1.34k op/s 2024-03-28T18:45:07.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:06 smithi145 ceph-mon[32151]: pgmap v571: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 720 MiB/s wr, 1.34k op/s 2024-03-28T18:45:09.280 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:08 smithi145 ceph-mon[32151]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 686 MiB/s wr, 1.26k op/s 2024-03-28T18:45:09.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:08 smithi192 ceph-mon[36589]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 686 MiB/s wr, 1.26k op/s 2024-03-28T18:45:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:08 smithi164 ceph-mon[36248]: pgmap v572: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 686 MiB/s wr, 1.26k op/s 2024-03-28T18:45:11.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:10 smithi145 ceph-mon[32151]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 734 MiB/s wr, 1.41k op/s 2024-03-28T18:45:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:10 smithi192 ceph-mon[36589]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 734 MiB/s wr, 1.41k op/s 2024-03-28T18:45:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:10 smithi164 ceph-mon[36248]: pgmap v573: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 734 MiB/s wr, 1.41k op/s 2024-03-28T18:45:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.605+0000 7fa10cdb9640 1 -- 172.21.15.145:0/1488977759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a6740 msgr2=0x7fa1000ab000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:12.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.605+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/1488977759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a6740 0x7fa1000ab000 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009900 tx=0x7fa0fc02bc60 comp rx=0 tx=0).stop 2024-03-28T18:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.609+0000 7fa10cdb9640 1 -- 172.21.15.145:0/1488977759 shutdown_connections 2024-03-28T18:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.609+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/1488977759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a6740 0x7fa1000ab000 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.609+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/1488977759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a5da0 0x7fa1000a6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.609+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/1488977759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1000a4c70 0x7fa1000a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.609+0000 7fa10cdb9640 1 -- 172.21.15.145:0/1488977759 >> 172.21.15.145:0/1488977759 conn(0x7fa1000a0100 msgr2=0x7fa1000a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.611+0000 7fa10cdb9640 1 -- 172.21.15.145:0/1488977759 shutdown_connections 2024-03-28T18:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.611+0000 7fa10cdb9640 1 -- 172.21.15.145:0/1488977759 wait complete. 2024-03-28T18:45:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.612+0000 7fa10cdb9640 1 Processor -- start 2024-03-28T18:45:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.612+0000 7fa10cdb9640 1 -- start start 2024-03-28T18:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.612+0000 7fa10cdb9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a4c70 0x7fa1001436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10cdb9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10cdb9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1000a6740 0x7fa10014aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10cdb9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000b9130 con 0x7fa1000a5da0 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10cdb9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000b8fb0 con 0x7fa1000a4c70 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10cdb9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000b92b0 con 0x7fa1000a6740 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa105d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa105d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:48820/0 (socket says 172.21.15.145:48820) 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa105d9b640 1 -- 172.21.15.145:0/2187802640 learned_addr learned my addr 172.21.15.145:0/2187802640 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.613+0000 7fa10659c640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a4c70 0x7fa1001436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 -- 172.21.15.145:0/2187802640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1000a6740 msgr2=0x7fa10014aca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1000a6740 0x7fa10014aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 -- 172.21.15.145:0/2187802640 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a4c70 msgr2=0x7fa1001436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a4c70 0x7fa1001436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 -- 172.21.15.145:0/2187802640 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10014b310 con 0x7fa1000a5da0 2024-03-28T18:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b450 tx=0x7fa0f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa0f77fe640 1 -- 172.21.15.145:0/2187802640 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8015020 con 0x7fa1000a5da0 2024-03-28T18:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa0f77fe640 1 -- 172.21.15.145:0/2187802640 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8011d70 con 0x7fa1000a5da0 2024-03-28T18:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.614+0000 7fa0f77fe640 1 -- 172.21.15.145:0/2187802640 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800f340 con 0x7fa1000a5da0 2024-03-28T18:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.615+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10014b600 con 0x7fa1000a5da0 2024-03-28T18:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.615+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10014bb00 con 0x7fa1000a5da0 2024-03-28T18:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.620+0000 7fa10f9a4740 1 Processor -- start 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 -- start start 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f9997010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9999b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f98f6b10 con 0x5594f9991b20 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f989f900 con 0x5594f9992510 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.621+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f98f2bb0 con 0x5594f9997680 2024-03-28T18:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.622+0000 7fa0f77fe640 1 -- 172.21.15.145:0/2187802640 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa0f8018020 con 0x7fa1000a5da0 2024-03-28T18:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.622+0000 7fa106d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.622+0000 7fa106d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:48834/0 (socket says 172.21.15.145:48834) 2024-03-28T18:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.622+0000 7fa106d9d640 1 -- 172.21.15.145:0/2990434608 learned_addr learned my addr 172.21.15.145:0/2990434608 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa0f77fe640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 0x7fa0f00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 -- 172.21.15.145:0/2990434608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 msgr2=0x5594f9999b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9999b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 -- 172.21.15.145:0/2990434608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 msgr2=0x5594f9997010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f9997010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 -- 172.21.15.145:0/2990434608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594f960b210 con 0x5594f9991b20 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa0f77fe640 1 -- 172.21.15.145:0/2187802640 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa0f80ce790 con 0x7fa1000a5da0 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.623+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 0x7fa0f00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.624+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc02efd0 tx=0x7fa0fc009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee69cc270e10bef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.624+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 0x7fa0f00a4ec0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc046000 tx=0x7fa0fc04d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.625+0000 7fa0f4ff9640 1 -- 172.21.15.145:0/2990434608 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc04d070 con 0x5594f9991b20 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.625+0000 7fa0f4ff9640 1 -- 172.21.15.145:0/2990434608 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc0054e0 con 0x5594f9991b20 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.626+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2990434608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 msgr2=0x5594f9991f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.626+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc02efd0 tx=0x7fa0fc009520 comp rx=0 tx=0).stop 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.630+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2990434608 shutdown_connections 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.630+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9999b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.630+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f9997010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.630+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2990434608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9991b20 0x5594f9991f40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.630+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2990434608 >> 172.21.15.145:0/2990434608 conn(0x5594f998d090 msgr2=0x5594f998d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.631+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2990434608 shutdown_connections 2024-03-28T18:45:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.632+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2990434608 wait complete. 2024-03-28T18:45:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 -- start start 2024-03-28T18:45:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 Processor -- start 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f999c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f999c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 0x5594f99a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.633+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f999a680 con 0x5594f9997680 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.634+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f98b2d50 con 0x5594f9992510 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.634+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594f999a800 con 0x5594f9991b20 2024-03-28T18:45:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.634+0000 7fa10f9a4740 10 client.0 _ll_register_callbacks cb 0x5594f98b2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:12.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.636+0000 7fa106d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 0x5594f99a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.636+0000 7fa106d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 0x5594f99a36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:48844/0 (socket says 172.21.15.145:48844) 2024-03-28T18:45:12.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.636+0000 7fa106d9d640 1 -- 172.21.15.145:0/1938197146 learned_addr learned my addr 172.21.15.145:0/1938197146 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.637+0000 7fa105d9b640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f999c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.643 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.640+0000 7fa105d9b640 1 -- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 msgr2=0x5594f999c130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:12.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.640+0000 7fa105d9b640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f999c130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.640+0000 7fa105d9b640 1 -- 172.21.15.145:0/1938197146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 msgr2=0x5594f99a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:12.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.640+0000 7fa105d9b640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 0x5594f99a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.640+0000 7fa105d9b640 1 -- 172.21.15.145:0/1938197146 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594f96d7620 con 0x5594f9992510 2024-03-28T18:45:12.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.644+0000 7fa105d9b640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f999c670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800be10 tx=0x7fa0f809abb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b83682d3284b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.644+0000 7fa0edffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:45:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.647+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f809ad20 con 0x5594f9992510 2024-03-28T18:45:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.647+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594f9904050 con 0x5594f9992510 2024-03-28T18:45:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.647+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa0f80b95a0 con 0x5594f9992510 2024-03-28T18:45:12.648 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.648+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0f809ee70 con 0x5594f9992510 2024-03-28T18:45:12.650 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.650+0000 7fa10f9a4740 10 client.25652 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:12.650 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.650+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5594f99b7b80 con 0x5594f9992510 2024-03-28T18:45:12.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.651+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5594f9904050 con 0x5594f9992510 2024-03-28T18:45:12.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.651+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa0f809a170 con 0x5594f9992510 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 10 client.25652 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 10 client.25652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 20 client.25652 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f99b9130 0x5594f99bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 4 client.25652 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.652+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594f98fe3a0 con 0x5594f99b9130 2024-03-28T18:45:12.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.653+0000 7fa10659c640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f99b9130 0x5594f99bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.653+0000 7fa10659c640 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f99b9130 0x5594f99bb540 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdc9194e441852a6 server_cookie=b11e9ffcdebb7901 in_seq=0 out_seq=0 2024-03-28T18:45:12.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.653+0000 7fa0edffb640 10 client.25652 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.657+0000 7fa0edffb640 1 -- 172.21.15.145:0/1938197146 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5594f98fe3a0 con 0x5594f99b9130 2024-03-28T18:45:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.657+0000 7fa0edffb640 10 client.25652 handle_command_reply: tid=0 2024-03-28T18:45:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.657+0000 7fa10f9a4740 1 client.25652 shutdown 2024-03-28T18:45:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.659+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 msgr2=0x5594f999c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:12.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.659+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f999c670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800be10 tx=0x7fa0f809abb0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 shutdown_connections 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f99b9130 0x5594f99bb540 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9997680 0x5594f99a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9992510 0x5594f999c670 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/1938197146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f999c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 >> 172.21.15.145:0/1938197146 conn(0x5594f998d090 msgr2=0x5594f998c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:12.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.660+0000 7fa0edffb640 0 client.25652 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:12.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.661+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 shutdown_connections 2024-03-28T18:45:12.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.661+0000 7fa10f9a4740 1 -- 172.21.15.145:0/1938197146 wait complete. 2024-03-28T18:45:12.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.661+0000 7fa10f9a4740 20 client.25652 trim_cache size 0 max 16384 2024-03-28T18:45:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 Processor -- start 2024-03-28T18:45:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 -- start start 2024-03-28T18:45:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f998ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 0x5594f998bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014d620 con 0x5594f9992510 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014efa0 con 0x5594f9997680 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014dc80 con 0x5594f9991b20 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f998ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10659c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f998ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50312/0 (socket says 172.21.15.145:50312) 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa10659c640 1 -- 172.21.15.145:0/3155970346 learned_addr learned my addr 172.21.15.145:0/3155970346 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:12.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.695+0000 7fa106d9d640 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 0x5594f998bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 -- 172.21.15.145:0/3155970346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 msgr2=0x5594f998ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f998ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 -- 172.21.15.145:0/3155970346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 msgr2=0x5594f99a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 -- 172.21.15.145:0/3155970346 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594f999a510 con 0x5594f9997680 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa106d9d640 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 0x5594f998bf40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005ab0 tx=0x7fa0fc02bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=717e41d2ea2fbca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.696+0000 7fa0effff640 1 -- 172.21.15.145:0/3155970346 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc003bb0 con 0x5594f9997680 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.697+0000 7fa0effff640 1 -- 172.21.15.145:0/3155970346 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc0054e0 con 0x5594f9997680 2024-03-28T18:45:12.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.697+0000 7fa0effff640 1 -- 172.21.15.145:0/3155970346 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc004dd0 con 0x5594f9997680 2024-03-28T18:45:12.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.698+0000 7fa10f9a4740 1 -- 172.21.15.145:0/3155970346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 msgr2=0x5594f998bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:12.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.698+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 0x5594f998bf40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005ab0 tx=0x7fa0fc02bc60 comp rx=0 tx=0).stop 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 -- 172.21.15.145:0/3155970346 shutdown_connections 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9997680 0x5594f998bf40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/3155970346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9991b20 0x5594f998ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 -- 172.21.15.145:0/3155970346 >> 172.21.15.145:0/3155970346 conn(0x5594f998d090 msgr2=0x5594f999c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.701+0000 7fa10f9a4740 1 -- 172.21.15.145:0/3155970346 shutdown_connections 2024-03-28T18:45:12.702 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.702+0000 7fa10f9a4740 1 -- 172.21.15.145:0/3155970346 wait complete. 2024-03-28T18:45:12.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.703+0000 7fa10f9a4740 1 -- start start 2024-03-28T18:45:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.704+0000 7fa10f9a4740 1 Processor -- start 2024-03-28T18:45:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.704+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 0x5594f99a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.704+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10f9a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9a12a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014e940 con 0x5594f9992510 2024-03-28T18:45:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10f9a4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10014e2e0 con 0x5594f9991b20 2024-03-28T18:45:12.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 0x5594f99a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 0x5594f99a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:39764/0 (socket says 172.21.15.145:39764) 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10659c640 1 -- 172.21.15.145:0/2999056301 learned_addr learned my addr 172.21.15.145:0/2999056301 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000b4b80 con 0x5594f9997680 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa10f9a4740 10 client.0 _ll_register_callbacks cb 0x7fa0f80968a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa106d9d640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9a12a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.705+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 -- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 msgr2=0x5594f9a12a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9a12a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 -- 172.21.15.145:0/2999056301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 msgr2=0x5594f99a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 0x5594f99a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 -- 172.21.15.145:0/2999056301 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594f9a12ff0 con 0x5594f9992510 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa105d9b640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a0b80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80976a0 tx=0x7fa0f8096bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f502dd5d4a1ea2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.706+0000 7fa0edffb640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.707+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f809ad20 con 0x5594f9992510 2024-03-28T18:45:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.707+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0fc0054e0 con 0x5594f9992510 2024-03-28T18:45:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.708+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa0f80cbf90 con 0x5594f9992510 2024-03-28T18:45:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.709+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0f80a34c0 con 0x5594f9992510 2024-03-28T18:45:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.709+0000 7fa10f9a4740 10 client.15876 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.709+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5594f9a142e0 con 0x5594f9992510 2024-03-28T18:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc0054e0 con 0x5594f9992510 2024-03-28T18:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fa0f8099cf0 con 0x5594f9992510 2024-03-28T18:45:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 10 client.15876 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 20 client.15876 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f9a14b00 0x5594f9a16f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 4 client.15876 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:45:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.711+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594f987c710 con 0x5594f9a14b00 2024-03-28T18:45:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.713+0000 7fa10659c640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f9a14b00 0x5594f9a16f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.720+0000 7fa10659c640 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f9a14b00 0x5594f9a16f20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4436e65ccfc138ce server_cookie=7d026dcd60e5ae60 in_seq=0 out_seq=0 2024-03-28T18:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:12.722+0000 7fa0edffb640 10 client.15876 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:13.264 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:12 smithi164 ceph-mon[36248]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 719 MiB/s wr, 1.32k op/s 2024-03-28T18:45:13.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:12 smithi145 ceph-mon[32151]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 719 MiB/s wr, 1.32k op/s 2024-03-28T18:45:13.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:12 smithi192 ceph-mon[36589]: pgmap v574: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 719 MiB/s wr, 1.32k op/s 2024-03-28T18:45:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.450+0000 7fa0edffb640 1 -- 172.21.15.145:0/2999056301 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594f987c710 con 0x5594f9a14b00 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.450+0000 7fa0edffb640 10 client.15876 handle_command_reply: tid=0 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.450+0000 7fa10f9a4740 1 client.15876 shutdown 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.451+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 msgr2=0x5594f99a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.451+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a0b80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80976a0 tx=0x7fa0f8096bb0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 shutdown_connections 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5594f9a14b00 0x5594f9a16f20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5594f9997680 0x5594f9a12a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5594f9992510 0x5594f99a0b80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 --2- 172.21.15.145:0/2999056301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5594f9991b20 0x5594f99a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 >> 172.21.15.145:0/2999056301 conn(0x5594f998d090 msgr2=0x5594f99902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa0edffb640 0 client.15876 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 shutdown_connections 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 1 -- 172.21.15.145:0/2999056301 wait complete. 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.452+0000 7fa10f9a4740 20 client.15876 trim_cache size 0 max 16384 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:45:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:45:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.455+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 msgr2=0x7fa0f00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.455+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 0x7fa0f00a4ec0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc046000 tx=0x7fa0fc04d000 comp rx=0 tx=0).stop 2024-03-28T18:45:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.456+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 msgr2=0x7fa100143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:14.457 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.456+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b450 tx=0x7fa0f800b910 comp rx=0 tx=0).stop 2024-03-28T18:45:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.478+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 shutdown_connections 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.478+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0f00a2a00 0x7fa0f00a4ec0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.478+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1000a6740 0x7fa10014aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.478+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1000a5da0 0x7fa100143c30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.479+0000 7fa10cdb9640 1 --2- 172.21.15.145:0/2187802640 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1000a4c70 0x7fa1001436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.479+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 >> 172.21.15.145:0/2187802640 conn(0x7fa1000a0100 msgr2=0x7fa1000a6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.481+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 shutdown_connections 2024-03-28T18:45:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:14.481+0000 7fa10cdb9640 1 -- 172.21.15.145:0/2187802640 wait complete. 2024-03-28T18:45:15.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:14 smithi145 ceph-mon[32151]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 650 MiB/s wr, 1.13k op/s 2024-03-28T18:45:15.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:14 smithi192 ceph-mon[36589]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 650 MiB/s wr, 1.13k op/s 2024-03-28T18:45:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:14 smithi164 ceph-mon[36248]: pgmap v575: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 650 MiB/s wr, 1.13k op/s 2024-03-28T18:45:17.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:17 smithi145 ceph-mon[32151]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 MiB/s wr, 1.51k op/s 2024-03-28T18:45:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:17 smithi192 ceph-mon[36589]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 MiB/s wr, 1.51k op/s 2024-03-28T18:45:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:17 smithi164 ceph-mon[36248]: pgmap v576: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 MiB/s wr, 1.51k op/s 2024-03-28T18:45:19.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:19 smithi145 ceph-mon[32151]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:45:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:19 smithi192 ceph-mon[36589]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:45:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:19 smithi164 ceph-mon[36248]: pgmap v577: 129 pgs: 129 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 720 MiB/s wr, 1.51k op/s 2024-03-28T18:45:21.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:21 smithi145 ceph-mon[32151]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.59k op/s 2024-03-28T18:45:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:21 smithi192 ceph-mon[36589]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.59k op/s 2024-03-28T18:45:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:21 smithi164 ceph-mon[36248]: pgmap v578: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.59k op/s 2024-03-28T18:45:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:23 smithi192 ceph-mon[36589]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 1.41k op/s 2024-03-28T18:45:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:23 smithi164 ceph-mon[36248]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 1.41k op/s 2024-03-28T18:45:23.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:23 smithi145 ceph-mon[32151]: pgmap v579: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 1.41k op/s 2024-03-28T18:45:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:25 smithi192 ceph-mon[36589]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 647 MiB/s wr, 1.27k op/s 2024-03-28T18:45:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:25 smithi164 ceph-mon[36248]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 647 MiB/s wr, 1.27k op/s 2024-03-28T18:45:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:25 smithi145 ceph-mon[32151]: pgmap v580: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 647 MiB/s wr, 1.27k op/s 2024-03-28T18:45:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:27 smithi192 ceph-mon[36589]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 824 MiB/s wr, 1.63k op/s 2024-03-28T18:45:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:27 smithi164 ceph-mon[36248]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 824 MiB/s wr, 1.63k op/s 2024-03-28T18:45:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:27 smithi145 ceph-mon[32151]: pgmap v581: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 824 MiB/s wr, 1.63k op/s 2024-03-28T18:45:27.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.700+0000 7fbb0fef1640 1 -- 172.21.15.145:0/3093693362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 msgr2=0x7fbb0807b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.700+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/3093693362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb0807b280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000afd0 tx=0x7fbb000085b0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.703+0000 7fbb0fef1640 1 -- 172.21.15.145:0/3093693362 shutdown_connections 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.703+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/3093693362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb0807b280 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.703+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/3093693362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb08076440 0x7fbb080788b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.703+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/3093693362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 0x7fbb08073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.703+0000 7fbb0fef1640 1 -- 172.21.15.145:0/3093693362 >> 172.21.15.145:0/3093693362 conn(0x7fbb0806e6e0 msgr2=0x7fbb08070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.704+0000 7fbb0fef1640 1 -- 172.21.15.145:0/3093693362 shutdown_connections 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.704+0000 7fbb0fef1640 1 -- 172.21.15.145:0/3093693362 wait complete. 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 Processor -- start 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 -- start start 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 0x7fbb081c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb08076440 0x7fbb081c0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0807eb80 con 0x7fbb08076440 2024-03-28T18:45:27.710 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0807ea00 con 0x7fbb08073560 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0fef1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0807ed00 con 0x7fbb08078df0 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0e48e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.705+0000 7fbb0e48e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40084/0 (socket says 172.21.15.145:40084) 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 -- 172.21.15.145:0/1120563382 learned_addr learned my addr 172.21.15.145:0/1120563382 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 0x7fbb081c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 -- 172.21.15.145:0/1120563382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 msgr2=0x7fbb081c05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 0x7fbb081c05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 -- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb08076440 msgr2=0x7fbb081c0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb08076440 0x7fbb081c0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.706+0000 7fbb0e48e640 1 -- 172.21.15.145:0/1120563382 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb081c7b20 con 0x7fbb08078df0 2024-03-28T18:45:27.711 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.708+0000 7fbb0e48e640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb00008be0 tx=0x7fbb00036bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.716+0000 7fbafeffd640 1 -- 172.21.15.145:0/1120563382 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00044020 con 0x7fbb08078df0 2024-03-28T18:45:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.716+0000 7fbafeffd640 1 -- 172.21.15.145:0/1120563382 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb0002dbe0 con 0x7fbb08078df0 2024-03-28T18:45:27.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.716+0000 7fbafeffd640 1 -- 172.21.15.145:0/1120563382 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00040a30 con 0x7fbb08078df0 2024-03-28T18:45:27.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.716+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb081c7d50 con 0x7fbb08078df0 2024-03-28T18:45:27.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.716+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb081c8250 con 0x7fbb08078df0 2024-03-28T18:45:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.720+0000 7fbb12bbc740 1 Processor -- start 2024-03-28T18:45:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.721+0000 7fbb12bbc740 1 -- start start 2024-03-28T18:45:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4bdfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4b44b10 con 0x563ee4be5480 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4aed900 con 0x563ee4bdf920 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4b40bb0 con 0x563ee4be0310 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbafeffd640 1 -- 172.21.15.145:0/1120563382 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbb00047070 con 0x7fbb08078df0 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40090/0 (socket says 172.21.15.145:40090) 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.722+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/1896099714 learned_addr learned my addr 172.21.15.145:0/1896099714 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.723+0000 7fbafeffd640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 0x7fbaf40a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.723+0000 7fbafeffd640 1 -- 172.21.15.145:0/1120563382 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbb000f1000 con 0x7fbb08078df0 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.724+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.724+0000 7fbb0e48e640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 0x7fbaf40a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.725+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/1896099714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 msgr2=0x563ee4bdfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.725+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4bdfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.725+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/1896099714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 msgr2=0x563ee4be7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.725+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.726+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/1896099714 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ee4859210 con 0x563ee4be0310 2024-03-28T18:45:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.729+0000 7fbb0e48e640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 0x7fbaf40a4f90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbb0002ded0 tx=0x7fbb000f1250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.730+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400b670 tx=0x7fbb04005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf98ee46c4269af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.736+0000 7fbaf3fff640 1 -- 172.21.15.145:0/1896099714 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0403b070 con 0x563ee4be0310 2024-03-28T18:45:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.742+0000 7fbaf3fff640 1 -- 172.21.15.145:0/1896099714 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb04014750 con 0x563ee4be0310 2024-03-28T18:45:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.742+0000 7fbaf3fff640 1 -- 172.21.15.145:0/1896099714 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0403eaa0 con 0x563ee4be0310 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 -- 172.21.15.145:0/1896099714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 msgr2=0x563ee4be4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400b670 tx=0x7fbb04005f90 comp rx=0 tx=0).stop 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 -- 172.21.15.145:0/1896099714 shutdown_connections 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4be0310 0x563ee4be4e10 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400b670 tx=0x7fbb04005f90 comp rx=0 tx=0).stop 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/1896099714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4bdfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.744+0000 7fbb12bbc740 1 -- 172.21.15.145:0/1896099714 >> 172.21.15.145:0/1896099714 conn(0x563ee4bdae90 msgr2=0x563ee4bdb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.753+0000 7fbb12bbc740 1 -- 172.21.15.145:0/1896099714 shutdown_connections 2024-03-28T18:45:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.753+0000 7fbb12bbc740 1 -- 172.21.15.145:0/1896099714 wait complete. 2024-03-28T18:45:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.753+0000 7fbb12bbc740 1 -- start start 2024-03-28T18:45:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.753+0000 7fbb12bbc740 1 Processor -- start 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4be9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4bea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bef370 0x563ee4bf1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4be8430 con 0x563ee4be5480 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4b00d50 con 0x563ee4bdf920 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee4be85b0 con 0x563ee4bef370 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.754+0000 7fbb12bbc740 10 client.0 _ll_register_callbacks cb 0x563ee4b00720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.755+0000 7fbb0d48c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4be9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.755+0000 7fbb0d48c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4be9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:53038/0 (socket says 172.21.15.145:53038) 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.755+0000 7fbb0d48c640 1 -- 172.21.15.145:0/2193961859 learned_addr learned my addr 172.21.15.145:0/2193961859 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.755+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bef370 0x563ee4bf1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.757+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4bea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.762+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bef370 msgr2=0x563ee4bf1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.762+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bef370 0x563ee4bf1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.762+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2193961859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 msgr2=0x563ee4be9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.762+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4be9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.762+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2193961859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ee4925620 con 0x563ee4be5480 2024-03-28T18:45:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.765+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4bea340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x563ee4be07d0 tx=0x7fbb0403ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3730d4c7f1366ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.767+0000 7fbaf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:45:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.767+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0403b070 con 0x563ee4be5480 2024-03-28T18:45:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.768+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ee4b52050 con 0x563ee4be5480 2024-03-28T18:45:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.768+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbb0403ed10 con 0x563ee4be5480 2024-03-28T18:45:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.769+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb04052050 con 0x563ee4be5480 2024-03-28T18:45:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.787+0000 7fbb12bbc740 10 client.15885 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.787+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563ee4bf4ad0 con 0x563ee4be5480 2024-03-28T18:45:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563ee4b52050 con 0x563ee4be5480 2024-03-28T18:45:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbb04009890 con 0x563ee4be5480 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 10 client.15885 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 20 client.15885 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4bf5e80 0x563ee4bf8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 4 client.15885 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.789+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ee4b4c3a0 con 0x563ee4bf5e80 2024-03-28T18:45:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.791+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4bf5e80 0x563ee4bf8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.805+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4bf5e80 0x563ee4bf8270 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f053b8f250014a4 server_cookie=76fc09f15df3369a in_seq=0 out_seq=0 2024-03-28T18:45:27.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.806+0000 7fbaf17fa640 10 client.15885 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.807+0000 7fbaf17fa640 1 -- 172.21.15.145:0/2193961859 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563ee4b4c3a0 con 0x563ee4bf5e80 2024-03-28T18:45:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.807+0000 7fbaf17fa640 10 client.15885 handle_command_reply: tid=0 2024-03-28T18:45:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.807+0000 7fbb12bbc740 1 client.15885 shutdown 2024-03-28T18:45:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.813+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 msgr2=0x563ee4bea340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.813+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4bea340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x563ee4be07d0 tx=0x7fbb0403ea10 comp rx=0 tx=0).stop 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 shutdown_connections 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4bf5e80 0x563ee4bf8270 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bef370 0x563ee4bf1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4bea340 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2193961859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bdf920 0x563ee4be9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 >> 172.21.15.145:0/2193961859 conn(0x563ee4bdae90 msgr2=0x563ee4bda020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.816+0000 7fbaf17fa640 0 client.15885 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.817+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 shutdown_connections 2024-03-28T18:45:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.821+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2193961859 wait complete. 2024-03-28T18:45:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.824+0000 7fbb12bbc740 20 client.15885 trim_cache size 0 max 16384 2024-03-28T18:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.876+0000 7fbb12bbc740 1 Processor -- start 2024-03-28T18:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.876+0000 7fbb12bbc740 1 -- start start 2024-03-28T18:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.877+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4bf7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.877+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.877+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4bea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.877+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec00fb40 con 0x563ee4be5480 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.877+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0114c0 con 0x563ee4bef370 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0101a0 con 0x563ee4bdf920 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:48472/0 (socket says 172.21.15.145:48472) 2024-03-28T18:45:27.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2236927462 learned_addr learned my addr 172.21.15.145:0/2236927462 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2236927462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 msgr2=0x563ee4bf7e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4bf7e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2236927462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 msgr2=0x563ee4bea340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.878+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4bea340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.879+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/2236927462 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ee4be4a20 con 0x563ee4be5480 2024-03-28T18:45:27.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.879+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb04004730 tx=0x7fbb04004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2123da0ced2fdfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.880+0000 7fbaf37fe640 1 -- 172.21.15.145:0/2236927462 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0403b040 con 0x563ee4be5480 2024-03-28T18:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.880+0000 7fbaf37fe640 1 -- 172.21.15.145:0/2236927462 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb04009260 con 0x563ee4be5480 2024-03-28T18:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.881+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2236927462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 msgr2=0x563ee4be9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.881+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb04004730 tx=0x7fbb04004370 comp rx=0 tx=0).stop 2024-03-28T18:45:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.883+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2236927462 shutdown_connections 2024-03-28T18:45:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.883+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4bea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.883+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4be9e00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.883+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/2236927462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4bf7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.883+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2236927462 >> 172.21.15.145:0/2236927462 conn(0x563ee4bdae90 msgr2=0x563ee4bdb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.887+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2236927462 shutdown_connections 2024-03-28T18:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.887+0000 7fbb12bbc740 1 -- 172.21.15.145:0/2236927462 wait complete. 2024-03-28T18:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.888+0000 7fbb12bbc740 1 -- start start 2024-03-28T18:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.888+0000 7fbb12bbc740 1 Processor -- start 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4c05e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4c073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec010e60 con 0x563ee4be5480 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec010800 con 0x563ee4bef370 2024-03-28T18:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec00dd00 con 0x563ee4bdf920 2024-03-28T18:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.889+0000 7fbb12bbc740 10 client.0 _ll_register_callbacks cb 0x7fbb040044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:48474/0 (socket says 172.21.15.145:48474) 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/156666314 learned_addr learned my addr 172.21.15.145:0/156666314 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 msgr2=0x563ee4c05e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4c05e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/156666314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 msgr2=0x563ee4c073d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4c073d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.892+0000 7fbb0dc8d640 1 -- 172.21.15.145:0/156666314 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ee4c079c0 con 0x563ee4be5480 2024-03-28T18:45:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.893+0000 7fbb0dc8d640 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbb04000c00 tx=0x7fbb040058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89eea76aff0068 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.893+0000 7fbaf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:45:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.894+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04005030 con 0x563ee4be5480 2024-03-28T18:45:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.894+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb04009260 con 0x563ee4be5480 2024-03-28T18:45:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.894+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbb040051d0 con 0x563ee4be5480 2024-03-28T18:45:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.895+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb04014e10 con 0x563ee4be5480 2024-03-28T18:45:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.898+0000 7fbb12bbc740 10 client.15891 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.898+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563ee4c50f10 con 0x563ee4be5480 2024-03-28T18:45:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.900+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb04009260 con 0x563ee4be5480 2024-03-28T18:45:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.900+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbb040035f0 con 0x563ee4be5480 2024-03-28T18:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 10 client.15891 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 20 client.15891 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4c51660 0x563ee4c53b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 4 client.15891 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.903+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563ee4aca710 con 0x563ee4c51660 2024-03-28T18:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.913+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4c51660 0x563ee4c53b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.941+0000 7fbb0d48c640 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4c51660 0x563ee4c53b20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=619e14801c33eb9 server_cookie=8e32963e7d83c6f8 in_seq=0 out_seq=0 2024-03-28T18:45:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:27.942+0000 7fbaf17fa640 10 client.15891 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:29.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:29 smithi192 ceph-mon[36589]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 686 MiB/s wr, 1.35k op/s 2024-03-28T18:45:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:29 smithi164 ceph-mon[36248]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 686 MiB/s wr, 1.35k op/s 2024-03-28T18:45:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:29 smithi145 ceph-mon[32151]: pgmap v582: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 686 MiB/s wr, 1.35k op/s 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.189+0000 7fbaf17fa640 1 -- 172.21.15.145:0/156666314 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563ee4aca710 con 0x563ee4c51660 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.189+0000 7fbaf17fa640 10 client.15891 handle_command_reply: tid=0 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.189+0000 7fbb12bbc740 1 client.15891 shutdown 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 msgr2=0x563ee4c063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbb04000c00 tx=0x7fbb040058c0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 shutdown_connections 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x563ee4c51660 0x563ee4c53b20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563ee4bef370 0x563ee4c073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x563ee4be5480 0x563ee4c063c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 --2- 172.21.15.145:0/156666314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563ee4bdf920 0x563ee4c05e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 >> 172.21.15.145:0/156666314 conn(0x563ee4bdae90 msgr2=0x563ee4bf7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbaf17fa640 0 client.15891 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 shutdown_connections 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 1 -- 172.21.15.145:0/156666314 wait complete. 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.190+0000 7fbb12bbc740 20 client.15891 trim_cache size 0 max 16384 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 msgr2=0x7fbaf40a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 0x7fbaf40a4f90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbb0002ded0 tx=0x7fbb000f1250 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 msgr2=0x7fbb081c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbb00008be0 tx=0x7fbb00036bf0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 shutdown_connections 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbaf40a2ad0 0x7fbaf40a4f90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.194+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08078df0 0x7fbb081c7450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.195+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb08076440 0x7fbb081c0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.195+0000 7fbb0fef1640 1 --2- 172.21.15.145:0/1120563382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb08073560 0x7fbb081c05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.195+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 >> 172.21.15.145:0/1120563382 conn(0x7fbb0806e6e0 msgr2=0x7fbb080785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.195+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 shutdown_connections 2024-03-28T18:45:31.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:31.195+0000 7fbb0fef1640 1 -- 172.21.15.145:0/1120563382 wait complete. 2024-03-28T18:45:31.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:31 smithi192 ceph-mon[36589]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.41k op/s 2024-03-28T18:45:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:31 smithi164 ceph-mon[36248]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.41k op/s 2024-03-28T18:45:31.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:31 smithi145 ceph-mon[32151]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 733 MiB/s wr, 1.41k op/s 2024-03-28T18:45:33.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:33 smithi192 ceph-mon[36589]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 725 MiB/s wr, 1.41k op/s 2024-03-28T18:45:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:33 smithi164 ceph-mon[36248]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 725 MiB/s wr, 1.41k op/s 2024-03-28T18:45:33.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:33 smithi145 ceph-mon[32151]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 725 MiB/s wr, 1.41k op/s 2024-03-28T18:45:35.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:35 smithi192 ceph-mon[36589]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 642 MiB/s wr, 1.24k op/s 2024-03-28T18:45:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:35 smithi164 ceph-mon[36248]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 642 MiB/s wr, 1.24k op/s 2024-03-28T18:45:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:35 smithi145 ceph-mon[32151]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 642 MiB/s wr, 1.24k op/s 2024-03-28T18:45:37.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:37 smithi192 ceph-mon[36589]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.44k op/s 2024-03-28T18:45:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:37 smithi164 ceph-mon[36248]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.44k op/s 2024-03-28T18:45:37.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:37 smithi145 ceph-mon[32151]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.44k op/s 2024-03-28T18:45:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:39 smithi192 ceph-mon[36589]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 682 MiB/s wr, 1.36k op/s 2024-03-28T18:45:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:39 smithi164 ceph-mon[36248]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 682 MiB/s wr, 1.36k op/s 2024-03-28T18:45:39.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:39 smithi145 ceph-mon[32151]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 682 MiB/s wr, 1.36k op/s 2024-03-28T18:45:41.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:41 smithi192 ceph-mon[36589]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.56k op/s 2024-03-28T18:45:41.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:41 smithi164 ceph-mon[36248]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.56k op/s 2024-03-28T18:45:41.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:41 smithi145 ceph-mon[32151]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.56k op/s 2024-03-28T18:45:43.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:43 smithi192 ceph-mon[36589]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 673 MiB/s wr, 1.47k op/s 2024-03-28T18:45:43.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:43 smithi164 ceph-mon[36248]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 673 MiB/s wr, 1.47k op/s 2024-03-28T18:45:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:43 smithi145 ceph-mon[32151]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 673 MiB/s wr, 1.47k op/s 2024-03-28T18:45:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.443+0000 7fc77a7a9640 1 -- 172.21.15.145:0/3508059041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc77410d9c0 msgr2=0x7fc77410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.448 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.443+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/3508059041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc77410d9c0 0x7fc77410de40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc768009900 tx=0x7fc76802bc60 comp rx=0 tx=0).stop 2024-03-28T18:45:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.452+0000 7fc77a7a9640 1 -- 172.21.15.145:0/3508059041 shutdown_connections 2024-03-28T18:45:44.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.452+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/3508059041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc77410d9c0 0x7fc77410de40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.452+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/3508059041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410b010 0x7fc77410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.452+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/3508059041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7741086a0 0x7fc77410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.452+0000 7fc77a7a9640 1 -- 172.21.15.145:0/3508059041 >> 172.21.15.145:0/3508059041 conn(0x7fc77406e6e0 msgr2=0x7fc774070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:44.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.457+0000 7fc77a7a9640 1 -- 172.21.15.145:0/3508059041 shutdown_connections 2024-03-28T18:45:44.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.459+0000 7fc77a7a9640 1 -- 172.21.15.145:0/3508059041 wait complete. 2024-03-28T18:45:44.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.460+0000 7fc77a7a9640 1 Processor -- start 2024-03-28T18:45:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.463+0000 7fc77a7a9640 1 -- start start 2024-03-28T18:45:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7741086a0 0x7fc774114cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 0x7fc7741151f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77411c220 con 0x7fc77410b010 2024-03-28T18:45:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77411c0a0 con 0x7fc77410d9c0 2024-03-28T18:45:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.464+0000 7fc77a7a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77411c3a0 con 0x7fc7741086a0 2024-03-28T18:45:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.466+0000 7fc778d46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.466+0000 7fc778d46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:38806/0 (socket says 172.21.15.145:38806) 2024-03-28T18:45:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.466+0000 7fc778d46640 1 -- 172.21.15.145:0/1331277622 learned_addr learned my addr 172.21.15.145:0/1331277622 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.466+0000 7fc773fff640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7741086a0 0x7fc774114cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc778d46640 1 -- 172.21.15.145:0/1331277622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7741086a0 msgr2=0x7fc774114cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc778d46640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7741086a0 0x7fc774114cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc7737fe640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 0x7fc7741151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc778d46640 1 -- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 msgr2=0x7fc7741151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc778d46640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 0x7fc7741151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc778d46640 1 -- 172.21.15.145:0/1331277622 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774118320 con 0x7fc77410d9c0 2024-03-28T18:45:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.467+0000 7fc7737fe640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 0x7fc7741151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:45:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.470+0000 7fc778d46640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc768009850 tx=0x7fc76802c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.471+0000 7fc7717fa640 1 -- 172.21.15.145:0/1331277622 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768041070 con 0x7fc77410d9c0 2024-03-28T18:45:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.471+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7741b6e70 con 0x7fc77410d9c0 2024-03-28T18:45:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.471+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7741b72d0 con 0x7fc77410d9c0 2024-03-28T18:45:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.473+0000 7fc7717fa640 1 -- 172.21.15.145:0/1331277622 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7680054e0 con 0x7fc77410d9c0 2024-03-28T18:45:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.473+0000 7fc7717fa640 1 -- 172.21.15.145:0/1331277622 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768034440 con 0x7fc77410d9c0 2024-03-28T18:45:44.479 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.477+0000 7fc77d392740 1 Processor -- start 2024-03-28T18:45:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.480+0000 7fc7717fa640 1 -- 172.21.15.145:0/1331277622 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc768044070 con 0x7fc77410d9c0 2024-03-28T18:45:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.481+0000 7fc7717fa640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 0x7fc75c0a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.481+0000 7fc7717fa640 1 -- 172.21.15.145:0/1331277622 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc7680f0c60 con 0x7fc77410d9c0 2024-03-28T18:45:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.481+0000 7fc77d392740 1 -- start start 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d508482d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508487e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5083e7b10 con 0x55d508483310 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d508390900 con 0x55d508482920 2024-03-28T18:45:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc77d392740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5083e3bb0 con 0x55d508488480 2024-03-28T18:45:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.482+0000 7fc778d46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.483+0000 7fc778d46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:36278/0 (socket says 172.21.15.145:36278) 2024-03-28T18:45:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.483+0000 7fc778d46640 1 -- 172.21.15.145:0/4178249707 learned_addr learned my addr 172.21.15.145:0/4178249707 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.484+0000 7fc778d46640 1 -- 172.21.15.145:0/4178249707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 msgr2=0x55d508482d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:45:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.484+0000 7fc778d46640 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d508482d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.484+0000 7fc778d46640 1 -- 172.21.15.145:0/4178249707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 msgr2=0x55d508487e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:45:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.484+0000 7fc778d46640 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508487e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.484+0000 7fc778d46640 1 -- 172.21.15.145:0/4178249707 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5080fc210 con 0x55d508488480 2024-03-28T18:45:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.487+0000 7fc7737fe640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 0x7fc75c0a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.488+0000 7fc7737fe640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 0x7fc75c0a4fa0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55d508483710 tx=0x7fc76c0073a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.489+0000 7fc778d46640 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc7680bcef0 tx=0x7fc7680bcdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76637fa7231a727 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.492+0000 7fc75a7fc640 1 -- 172.21.15.145:0/4178249707 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76804b040 con 0x55d508488480 2024-03-28T18:45:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.492+0000 7fc75a7fc640 1 -- 172.21.15.145:0/4178249707 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7680c1120 con 0x55d508488480 2024-03-28T18:45:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.493+0000 7fc77d392740 1 -- 172.21.15.145:0/4178249707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 msgr2=0x55d50848a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.493+0000 7fc77d392740 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc7680bcef0 tx=0x7fc7680bcdf0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.496+0000 7fc77d392740 1 -- 172.21.15.145:0/4178249707 shutdown_connections 2024-03-28T18:45:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.496+0000 7fc77d392740 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d50848a910 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.496+0000 7fc77d392740 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508487e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.496+0000 7fc77d392740 1 --2- 172.21.15.145:0/4178249707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d508482d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.496+0000 7fc77d392740 1 -- 172.21.15.145:0/4178249707 >> 172.21.15.145:0/4178249707 conn(0x55d50847de90 msgr2=0x55d50847e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:44.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.497+0000 7fc77d392740 1 -- 172.21.15.145:0/4178249707 shutdown_connections 2024-03-28T18:45:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.499+0000 7fc77d392740 1 -- 172.21.15.145:0/4178249707 wait complete. 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.500+0000 7fc77d392740 1 -- start start 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.500+0000 7fc77d392740 1 Processor -- start 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 0x55d50848c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d50848ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 0x55d508493e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d50848b1f0 con 0x55d508482920 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5083a3d50 con 0x55d508488480 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d50848b370 con 0x55d508483310 2024-03-28T18:45:44.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc77d392740 10 client.0 _ll_register_callbacks cb 0x55d5083a3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc773fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 0x55d50848c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc773fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 0x55d50848c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:56080/0 (socket says 172.21.15.145:56080) 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc773fff640 1 -- 172.21.15.145:0/1578311039 learned_addr learned my addr 172.21.15.145:0/1578311039 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc778d46640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 0x55d508493e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc778d46640 1 -- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 msgr2=0x55d50848ce40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc778d46640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d50848ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.501+0000 7fc778d46640 1 -- 172.21.15.145:0/1578311039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 msgr2=0x55d50848c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.502+0000 7fc778d46640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 0x55d50848c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.502+0000 7fc778d46640 1 -- 172.21.15.145:0/1578311039 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5081c8620 con 0x55d508488480 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.502+0000 7fc778d46640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 0x55d508493e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc76802c7e0 tx=0x7fc76802bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f04913f4ddcbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.503+0000 7fc7537fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.503+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76804b070 con 0x55d508488480 2024-03-28T18:45:44.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.503+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc768003800 con 0x55d508488480 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.508+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5083f5050 con 0x55d508488480 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.508+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7680c5990 con 0x55d508488480 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.508+0000 7fc77d392740 10 client.25673 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.508+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d50848b1f0 con 0x55d508488480 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.508+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d5083f5050 con 0x55d508488480 2024-03-28T18:45:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc7680bc0b0 con 0x55d508488480 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 10 client.25673 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 10 client.25673 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 20 client.25673 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084988b0 0x55d50849acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 4 client.25673 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.509+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5083ef3a0 con 0x55d5084988b0 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.510+0000 7fc773fff640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084988b0 0x55d50849acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.511+0000 7fc773fff640 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084988b0 0x55d50849acf0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62642681b0a9131a server_cookie=9a75b2f7609ccc06 in_seq=0 out_seq=0 2024-03-28T18:45:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.511+0000 7fc7537fe640 10 client.25673 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.514+0000 7fc7537fe640 1 -- 172.21.15.145:0/1578311039 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d5083ef3a0 con 0x55d5084988b0 2024-03-28T18:45:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.514+0000 7fc7537fe640 10 client.25673 handle_command_reply: tid=0 2024-03-28T18:45:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.514+0000 7fc77d392740 1 client.25673 shutdown 2024-03-28T18:45:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.516+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 msgr2=0x55d508493e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.516+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 0x55d508493e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc76802c7e0 tx=0x7fc76802bc60 comp rx=0 tx=0).stop 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 shutdown_connections 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084988b0 0x55d50849acf0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508488480 0x55d508493e80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d50848ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 --2- 172.21.15.145:0/1578311039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508482920 0x55d50848c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 >> 172.21.15.145:0/1578311039 conn(0x55d50847de90 msgr2=0x55d50847d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.518+0000 7fc7537fe640 0 client.25673 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:44.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.521+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 shutdown_connections 2024-03-28T18:45:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.527+0000 7fc77d392740 1 -- 172.21.15.145:0/1578311039 wait complete. 2024-03-28T18:45:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.528+0000 7fc77d392740 20 client.25673 trim_cache size 0 max 16384 2024-03-28T18:45:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 Processor -- start 2024-03-28T18:45:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 -- start start 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508493e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d508478eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76000fb30 con 0x55d508483310 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.573+0000 7fc77d392740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600114b0 con 0x55d508482920 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:38844/0 (socket says 172.21.15.145:38844) 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 -- 172.21.15.145:0/1795111890 learned_addr learned my addr 172.21.15.145:0/1795111890 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760010190 con 0x55d508488480 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc7737fe640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508493e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc778d46640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d508478eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 -- 172.21.15.145:0/1795111890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 msgr2=0x55d508478eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d508478eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 -- 172.21.15.145:0/1795111890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 msgr2=0x55d508493e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508493e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 -- 172.21.15.145:0/1795111890 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d508483fe0 con 0x55d508482920 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc778d46640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d508478eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.574+0000 7fc773fff640 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc764000c00 tx=0x7fc76402ddd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3adac648f6eac2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.575+0000 7fc759ffb640 1 -- 172.21.15.145:0/1795111890 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764030040 con 0x55d508482920 2024-03-28T18:45:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.575+0000 7fc759ffb640 1 -- 172.21.15.145:0/1795111890 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc76403b150 con 0x55d508482920 2024-03-28T18:45:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.575+0000 7fc759ffb640 1 -- 172.21.15.145:0/1795111890 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764036720 con 0x55d508482920 2024-03-28T18:45:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 msgr2=0x55d50847e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc764000c00 tx=0x7fc76402ddd0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 shutdown_connections 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508488480 0x55d508478eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508483310 0x55d508493e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 --2- 172.21.15.145:0/1795111890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d50847e270 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.576+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 >> 172.21.15.145:0/1795111890 conn(0x55d50847de90 msgr2=0x55d50848b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.577+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 shutdown_connections 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.577+0000 7fc77d392740 1 -- 172.21.15.145:0/1795111890 wait complete. 2024-03-28T18:45:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.577+0000 7fc77d392740 1 -- start start 2024-03-28T18:45:44.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.578+0000 7fc77d392740 1 Processor -- start 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.578+0000 7fc77d392740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d5084909a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.578+0000 7fc77d392740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d508490ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.578+0000 7fc77d392740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 0x55d5084f2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.579+0000 7fc77d392740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760010e50 con 0x55d508488480 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.579+0000 7fc77d392740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600107f0 con 0x55d508482920 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.579+0000 7fc77d392740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76000dcf0 con 0x55d508483310 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.579+0000 7fc77d392740 10 client.0 _ll_register_callbacks cb 0x7fc7680ee620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.580+0000 7fc7737fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d508490ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.580+0000 7fc773fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d5084909a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.580+0000 7fc7737fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d508490ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:36284/0 (socket says 172.21.15.145:36284) 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.580+0000 7fc7737fe640 1 -- 172.21.15.145:0/2476833399 learned_addr learned my addr 172.21.15.145:0/2476833399 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.580+0000 7fc778d46640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 0x55d5084f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.581+0000 7fc778d46640 1 -- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 msgr2=0x55d508490ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.581+0000 7fc778d46640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d508490ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.581+0000 7fc778d46640 1 -- 172.21.15.145:0/2476833399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 msgr2=0x55d5084909a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.581+0000 7fc778d46640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d5084909a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.581+0000 7fc778d46640 1 -- 172.21.15.145:0/2476833399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d508493f80 con 0x55d508488480 2024-03-28T18:45:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.582+0000 7fc778d46640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 0x55d5084f2170 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc7680bdaa0 tx=0x7fc768005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446a14ae0f40512 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.582+0000 7fc7537fe640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:45:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.583+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76804b040 con 0x55d508488480 2024-03-28T18:45:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.583+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc7680c5be0 con 0x55d508488480 2024-03-28T18:45:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.583+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc76403b150 con 0x55d508488480 2024-03-28T18:45:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7680b9a00 con 0x55d508488480 2024-03-28T18:45:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 10 client.15900 fetch_fsmap learned FSMap version 31 2024-03-28T18:45:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d508493f80 con 0x55d508488480 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7680cc030 con 0x55d508488480 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fc7680bc750 con 0x55d508488480 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 10 client.15900 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 20 client.15900 populate_metadata read hostname 'smithi145' 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084f3b10 0x55d5084f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 4 client.15900 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7fc76403b150 con 0x55d5084f3b10 2024-03-28T18:45:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.584+0000 7fc773fff640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084f3b10 0x55d5084f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.586+0000 7fc7537fe640 10 client.15900 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:44.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:44.586+0000 7fc773fff640 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084f3b10 0x55d5084f5f30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b68f39e2101ebae server_cookie=130bd60b0ff3d55e in_seq=0 out_seq=0 2024-03-28T18:45:45.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:45 smithi192 ceph-mon[36589]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 603 MiB/s wr, 1.29k op/s 2024-03-28T18:45:45.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:45 smithi164 ceph-mon[36248]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 603 MiB/s wr, 1.29k op/s 2024-03-28T18:45:45.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:45 smithi145 ceph-mon[32151]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 603 MiB/s wr, 1.29k op/s 2024-03-28T18:45:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.178+0000 7fc7537fe640 1 -- 172.21.15.145:0/2476833399 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7fc76403b150 con 0x55d5084f3b10 2024-03-28T18:45:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.178+0000 7fc7537fe640 10 client.15900 handle_command_reply: tid=0 2024-03-28T18:45:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.178+0000 7fc77d392740 1 client.15900 shutdown 2024-03-28T18:45:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.178+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 msgr2=0x55d5084f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.178+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 0x55d5084f2170 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc7680bdaa0 tx=0x7fc768005fd0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 shutdown_connections 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55d5084f3b10 0x55d5084f5f30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d508488480 0x55d5084f2170 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d508483310 0x55d508490ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 --2- 172.21.15.145:0/2476833399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d508482920 0x55d5084909a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 >> 172.21.15.145:0/2476833399 conn(0x55d50847de90 msgr2=0x55d508478eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc7537fe640 0 client.15900 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 shutdown_connections 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.179+0000 7fc77d392740 1 -- 172.21.15.145:0/2476833399 wait complete. 2024-03-28T18:45:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.180+0000 7fc77d392740 20 client.15900 trim_cache size 0 max 16384 2024-03-28T18:45:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:45:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:45:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:45:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:45:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 msgr2=0x7fc75c0a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 0x7fc75c0a4fa0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55d508483710 tx=0x7fc76c0073a0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 msgr2=0x7fc7741b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc768009850 tx=0x7fc76802c690 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 shutdown_connections 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77410d9c0 0x7fc7741b6930 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc77410b010 0x7fc7741151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fc75c0a2ae0 0x7fc75c0a4fa0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 --2- 172.21.15.145:0/1331277622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7741086a0 0x7fc774114cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.193+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 >> 172.21.15.145:0/1331277622 conn(0x7fc77406e6e0 msgr2=0x7fc77410be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.194+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 shutdown_connections 2024-03-28T18:45:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:45:47.195+0000 7fc77a7a9640 1 -- 172.21.15.145:0/1331277622 wait complete. 2024-03-28T18:45:47.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:47 smithi192 ceph-mon[36589]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 751 MiB/s wr, 1.39k op/s 2024-03-28T18:45:47.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:47 smithi164 ceph-mon[36248]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 751 MiB/s wr, 1.39k op/s 2024-03-28T18:45:47.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:47 smithi145 ceph-mon[32151]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 751 MiB/s wr, 1.39k op/s 2024-03-28T18:45:49.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:49 smithi164 ceph-mon[36248]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.25k op/s 2024-03-28T18:45:49.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:49 smithi145 ceph-mon[32151]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.25k op/s 2024-03-28T18:45:49.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:49 smithi192 ceph-mon[36589]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.25k op/s 2024-03-28T18:45:50.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:50 smithi164 ceph-mon[36248]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.26k op/s 2024-03-28T18:45:50.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:50 smithi145 ceph-mon[32151]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.26k op/s 2024-03-28T18:45:50.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:50 smithi192 ceph-mon[36589]: pgmap v593: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 1.26k op/s 2024-03-28T18:45:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:52 smithi164 ceph-mon[36248]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 644 MiB/s wr, 996 op/s 2024-03-28T18:45:53.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:52 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:52 smithi145 ceph-mon[32151]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 644 MiB/s wr, 996 op/s 2024-03-28T18:45:53.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:52 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:52 smithi192 ceph-mon[36589]: pgmap v594: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 644 MiB/s wr, 996 op/s 2024-03-28T18:45:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:52 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 642 MiB/s wr, 1.02k op/s 2024-03-28T18:45:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: Unable to set osd_memory_target on smithi192 to 182616985: error parsing value: Value '182616985' is below minimum 939524096 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:55.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 642 MiB/s wr, 1.02k op/s 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: Unable to set osd_memory_target on smithi192 to 182616985: error parsing value: Value '182616985' is below minimum 939524096 2024-03-28T18:45:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:55.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:55.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: pgmap v595: 129 pgs: 129 active+clean; 10 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 642 MiB/s wr, 1.02k op/s 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: Unable to set osd_memory_target on smithi192 to 182616985: error parsing value: Value '182616985' is below minimum 939524096 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:55.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:45:57.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:56 smithi164 ceph-mon[36248]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.53k op/s 2024-03-28T18:45:57.278 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:56 smithi145 ceph-mon[32151]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.53k op/s 2024-03-28T18:45:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:56 smithi192 ceph-mon[36589]: pgmap v596: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 761 MiB/s wr, 1.53k op/s 2024-03-28T18:45:59.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:45:58 smithi164 ceph-mon[36248]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 638 MiB/s wr, 1.53k op/s 2024-03-28T18:45:59.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:45:58 smithi145 ceph-mon[32151]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 638 MiB/s wr, 1.53k op/s 2024-03-28T18:45:59.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:45:58 smithi192 ceph-mon[36589]: pgmap v597: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 638 MiB/s wr, 1.53k op/s 2024-03-28T18:46:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.487+0000 7ff8a9962640 1 -- 172.21.15.145:0/2784529846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a41086a0 msgr2=0x7ff8a410aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.487+0000 7ff8a9962640 1 --2- 172.21.15.145:0/2784529846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a41086a0 0x7ff8a410aad0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009ca0 tx=0x7ff89c02c120 comp rx=0 tx=0).stop 2024-03-28T18:46:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.489+0000 7ff8a9962640 1 -- 172.21.15.145:0/2784529846 shutdown_connections 2024-03-28T18:46:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.489+0000 7ff8a9962640 1 --2- 172.21.15.145:0/2784529846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 0x7ff8a410de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.489+0000 7ff8a9962640 1 --2- 172.21.15.145:0/2784529846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a410b010 0x7ff8a410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.489+0000 7ff8a9962640 1 --2- 172.21.15.145:0/2784529846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a41086a0 0x7ff8a410aad0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.489+0000 7ff8a9962640 1 -- 172.21.15.145:0/2784529846 >> 172.21.15.145:0/2784529846 conn(0x7ff8a406e6e0 msgr2=0x7ff8a4070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.496+0000 7ff8a9962640 1 -- 172.21.15.145:0/2784529846 shutdown_connections 2024-03-28T18:46:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.502+0000 7ff8a9962640 1 -- 172.21.15.145:0/2784529846 wait complete. 2024-03-28T18:46:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.503+0000 7ff8a9962640 1 Processor -- start 2024-03-28T18:46:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 -- start start 2024-03-28T18:46:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a41086a0 0x7ff8a4114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 0x7ff8a41b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a411c1f0 con 0x7ff8a410b010 2024-03-28T18:46:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a411c070 con 0x7ff8a410d9c0 2024-03-28T18:46:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.504+0000 7ff8a9962640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a411c370 con 0x7ff8a41086a0 2024-03-28T18:46:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.506+0000 7ff8a27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.506+0000 7ff8a27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52356/0 (socket says 172.21.15.145:52356) 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.506+0000 7ff8a27fc640 1 -- 172.21.15.145:0/3062515625 learned_addr learned my addr 172.21.15.145:0/3062515625 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.507+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 0x7ff8a41b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.508+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a41086a0 0x7ff8a4114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.509+0000 7ff8a27fc640 1 -- 172.21.15.145:0/3062515625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a41086a0 msgr2=0x7ff8a4114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.509+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a41086a0 0x7ff8a4114cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.509+0000 7ff8a27fc640 1 -- 172.21.15.145:0/3062515625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 msgr2=0x7ff8a41b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.509+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 0x7ff8a41b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.509+0000 7ff8a27fc640 1 -- 172.21.15.145:0/3062515625 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a4118390 con 0x7ff8a410b010 2024-03-28T18:46:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.510+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff89800b450 tx=0x7ff89800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.525+0000 7ff8a8960640 1 -- 172.21.15.145:0/3062515625 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898015020 con 0x7ff8a410b010 2024-03-28T18:46:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.525+0000 7ff8a8960640 1 -- 172.21.15.145:0/3062515625 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff898011d70 con 0x7ff8a410b010 2024-03-28T18:46:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.525+0000 7ff8a8960640 1 -- 172.21.15.145:0/3062515625 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800f340 con 0x7ff8a410b010 2024-03-28T18:46:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.526+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a41b6f00 con 0x7ff8a410b010 2024-03-28T18:46:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.526+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a41b7390 con 0x7ff8a410b010 2024-03-28T18:46:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.533+0000 7ff8ac62f740 1 Processor -- start 2024-03-28T18:46:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 -- start start 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab381f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab381fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381fd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab3815ab10 con 0x55ab381f5920 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab38103900 con 0x55ab381fb480 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab38156bb0 con 0x55ab381f6310 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab381f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab381f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:52364/0 (socket says 172.21.15.145:52364) 2024-03-28T18:46:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.543+0000 7ff8a37fe640 1 -- 172.21.15.145:0/3247240167 learned_addr learned my addr 172.21.15.145:0/3247240167 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.544+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab381fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.544+0000 7ff8a8960640 1 -- 172.21.15.145:0/3062515625 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff898018020 con 0x7ff8a410b010 2024-03-28T18:46:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.545+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381fd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.545+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/3247240167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 msgr2=0x55ab381fd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.545+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381fd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.545+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/3247240167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 msgr2=0x55ab381f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.545+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab381f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.546+0000 7ff8a8960640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 0x7ff88c0a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.546+0000 7ff8a8960640 1 -- 172.21.15.145:0/3062515625 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff8980ce6c0 con 0x7ff8a410b010 2024-03-28T18:46:00.553 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.552+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/3247240167 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab37e6f210 con 0x55ab381f6310 2024-03-28T18:46:00.553 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.552+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:00.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.554+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab381fae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009bf0 tx=0x7ff89c0098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc1bc57e561a4455 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.555+0000 7ff889ffb640 1 -- 172.21.15.145:0/3247240167 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c043020 con 0x55ab381f6310 2024-03-28T18:46:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.556+0000 7ff889ffb640 1 -- 172.21.15.145:0/3247240167 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c003970 con 0x55ab381f6310 2024-03-28T18:46:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.556+0000 7ff889ffb640 1 -- 172.21.15.145:0/3247240167 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c039040 con 0x55ab381f6310 2024-03-28T18:46:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 -- 172.21.15.145:0/3247240167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 msgr2=0x55ab381fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab381fae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009bf0 tx=0x7ff89c0098b0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 -- 172.21.15.145:0/3247240167 shutdown_connections 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab381fae10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/3247240167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab381f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 -- 172.21.15.145:0/3247240167 >> 172.21.15.145:0/3247240167 conn(0x55ab381f0e90 msgr2=0x55ab381f1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:00.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 -- 172.21.15.145:0/3247240167 shutdown_connections 2024-03-28T18:46:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.557+0000 7ff8ac62f740 1 -- 172.21.15.145:0/3247240167 wait complete. 2024-03-28T18:46:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.558+0000 7ff8ac62f740 1 -- start start 2024-03-28T18:46:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.559+0000 7ff8ac62f740 1 Processor -- start 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.564+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 0x7ff88c0a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab38200170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381fb480 0x55ab382071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab381fe420 con 0x55ab381fb480 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab38116d50 con 0x55ab381f5920 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab381fe5a0 con 0x55ab381f6310 2024-03-28T18:46:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.565+0000 7ff8ac62f740 10 client.0 _ll_register_callbacks cb 0x55ab38116720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:00.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.566+0000 7ff8a27fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab38200170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.566+0000 7ff8a2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.567+0000 7ff8a2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:56654/0 (socket says 172.21.15.145:56654) 2024-03-28T18:46:00.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.567+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2928893973 learned_addr learned my addr 172.21.15.145:0/2928893973 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.568+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 msgr2=0x55ab38200170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.568+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab38200170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.569+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2928893973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381fb480 msgr2=0x55ab382071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.569+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381fb480 0x55ab382071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.569+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2928893973 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab37f3b620 con 0x55ab381f5920 2024-03-28T18:46:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.575+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff89c037040 tx=0x7ff89c032d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2b2672f7274e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.576+0000 7ff882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:00.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.576+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c043030 con 0x55ab381f5920 2024-03-28T18:46:00.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.580+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 0x7ff88c0a4fa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55ab381f6b70 tx=0x7ff8940098b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.580+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff89c0042f0 con 0x55ab381f5920 2024-03-28T18:46:00.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.580+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff89c003970 con 0x55ab381f5920 2024-03-28T18:46:00.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.581+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff89c058050 con 0x55ab381f5920 2024-03-28T18:46:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.584+0000 7ff8ac62f740 10 client.25691 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:00.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.584+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ab3820a520 con 0x55ab381f5920 2024-03-28T18:46:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.589+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c003970 con 0x55ab381f5920 2024-03-28T18:46:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.591+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff89c03fc30 con 0x55ab381f5920 2024-03-28T18:46:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 10 client.25691 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 10 client.25691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 20 client.25691 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab3820b870 0x55ab3820dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 4 client.25691 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.593+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab38168050 con 0x55ab3820b870 2024-03-28T18:46:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.596+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab3820b870 0x55ab3820dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.605+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab3820b870 0x55ab3820dc60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bfcaec7a47ca53d server_cookie=3974deeae9db6f2a in_seq=0 out_seq=0 2024-03-28T18:46:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.605+0000 7ff882ffd640 10 client.25691 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.611+0000 7ff882ffd640 1 -- 172.21.15.145:0/2928893973 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ab38168050 con 0x55ab3820b870 2024-03-28T18:46:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.611+0000 7ff882ffd640 10 client.25691 handle_command_reply: tid=0 2024-03-28T18:46:00.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.613+0000 7ff8ac62f740 1 client.25691 shutdown 2024-03-28T18:46:00.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 msgr2=0x55ab381ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff89c037040 tx=0x7ff89c032d80 comp rx=0 tx=0).stop 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 shutdown_connections 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab3820b870 0x55ab3820dc60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381fb480 0x55ab382071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab38200170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2928893973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381f5920 0x55ab381ffc30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.616+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 >> 172.21.15.145:0/2928893973 conn(0x55ab381f0e90 msgr2=0x55ab381f0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.617+0000 7ff882ffd640 0 client.25691 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.617+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 shutdown_connections 2024-03-28T18:46:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.617+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2928893973 wait complete. 2024-03-28T18:46:00.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.617+0000 7ff8ac62f740 20 client.25691 trim_cache size 0 max 16384 2024-03-28T18:46:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.656+0000 7ff8ac62f740 1 Processor -- start 2024-03-28T18:46:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.657+0000 7ff8ac62f740 1 -- start start 2024-03-28T18:46:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f6310 0x55ab38207140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41c16c0 con 0x55ab381f6310 2024-03-28T18:46:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41c3040 con 0x55ab381fb480 2024-03-28T18:46:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41c1d20 con 0x55ab381f5920 2024-03-28T18:46:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.658+0000 7ff8a27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f6310 0x55ab38207140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.659+0000 7ff8a2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.659+0000 7ff8a2ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:44120/0 (socket says 172.21.15.145:44120) 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.659+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2408047850 learned_addr learned my addr 172.21.15.145:0/2408047850 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.659+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.660+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2408047850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 msgr2=0x55ab381efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.660+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381efcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.660+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2408047850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f6310 msgr2=0x55ab38207140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.660+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f6310 0x55ab38207140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.660+0000 7ff8a2ffd640 1 -- 172.21.15.145:0/2408047850 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab3820e240 con 0x55ab381f5920 2024-03-28T18:46:00.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.661+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff89c02c160 tx=0x7ff89c009890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc895b22095a3088 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.664+0000 7ff8897fa640 1 -- 172.21.15.145:0/2408047850 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c043030 con 0x55ab381f5920 2024-03-28T18:46:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.664+0000 7ff8897fa640 1 -- 172.21.15.145:0/2408047850 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c003970 con 0x55ab381f5920 2024-03-28T18:46:00.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.664+0000 7ff8897fa640 1 -- 172.21.15.145:0/2408047850 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c039050 con 0x55ab381f5920 2024-03-28T18:46:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.666+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2408047850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 msgr2=0x55ab381ef790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.666+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff89c02c160 tx=0x7ff89c009890 comp rx=0 tx=0).stop 2024-03-28T18:46:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.667+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2408047850 shutdown_connections 2024-03-28T18:46:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.668+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab381efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.668+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f6310 0x55ab38207140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.668+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2408047850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f5920 0x55ab381ef790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.668+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2408047850 >> 172.21.15.145:0/2408047850 conn(0x55ab381f0e90 msgr2=0x55ab381f1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.672+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2408047850 shutdown_connections 2024-03-28T18:46:00.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.672+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2408047850 wait complete. 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 -- start start 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 Processor -- start 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab382043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab382048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41c29e0 con 0x55ab381f5920 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41c2380 con 0x55ab381fb480 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41bf930 con 0x55ab381f6310 2024-03-28T18:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.673+0000 7ff8ac62f740 10 client.0 _ll_register_callbacks cb 0x7ff89c032aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.674+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.675+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:56680/0 (socket says 172.21.15.145:56680) 2024-03-28T18:46:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.675+0000 7ff8a37fe640 1 -- 172.21.15.145:0/2152878018 learned_addr learned my addr 172.21.15.145:0/2152878018 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.676+0000 7ff8a27fc640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab382048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.677+0000 7ff8a37fe640 1 -- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 msgr2=0x55ab382048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.677+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab382048f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.677 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.677+0000 7ff8a37fe640 1 -- 172.21.15.145:0/2152878018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 msgr2=0x55ab382043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.677+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab382043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.677+0000 7ff8a37fe640 1 -- 172.21.15.145:0/2152878018 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab38205ef0 con 0x55ab381fb480 2024-03-28T18:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.679+0000 7ff8a37fe640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff894005fd0 tx=0x7ff894004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ffb340a6a4025a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.679+0000 7ff882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.682+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89402d060 con 0x55ab381fb480 2024-03-28T18:46:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.682+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff89c003970 con 0x55ab381fb480 2024-03-28T18:46:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.682+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff894018030 con 0x55ab381fb480 2024-03-28T18:46:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.683+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff894034050 con 0x55ab381fb480 2024-03-28T18:46:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 10 client.25694 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ab38207270 con 0x55ab381fb480 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c003970 con 0x55ab381fb480 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff894028be0 con 0x55ab381fb480 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 10 client.25694 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 10 client.25694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 20 client.25694 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab382675d0 0x55ab382699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 4 client.25694 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:46:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.684+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ab381623a0 con 0x55ab382675d0 2024-03-28T18:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.689+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab382675d0 0x55ab382699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.696+0000 7ff8a2ffd640 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab382675d0 0x55ab382699f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31decbfb6cc95fa7 server_cookie=fff59fe2add5394e in_seq=0 out_seq=0 2024-03-28T18:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:00.696+0000 7ff882ffd640 10 client.25694 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:01.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:00 smithi145 ceph-mon[32151]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.51k op/s 2024-03-28T18:46:01.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:00 smithi192 ceph-mon[36589]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.51k op/s 2024-03-28T18:46:01.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:00 smithi164 ceph-mon[36248]: pgmap v598: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 680 MiB/s wr, 1.51k op/s 2024-03-28T18:46:03.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:02 smithi145 ceph-mon[32151]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 603 MiB/s wr, 1.25k op/s 2024-03-28T18:46:03.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:02 smithi192 ceph-mon[36589]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 603 MiB/s wr, 1.25k op/s 2024-03-28T18:46:03.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:02 smithi164 ceph-mon[36248]: pgmap v599: 129 pgs: 129 active+clean; 10 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 603 MiB/s wr, 1.25k op/s 2024-03-28T18:46:05.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:04 smithi145 ceph-mon[32151]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:46:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff882ffd640 1 -- 172.21.15.145:0/2152878018 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ab381623a0 con 0x55ab382675d0 2024-03-28T18:46:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff882ffd640 10 client.25694 handle_command_reply: tid=0 2024-03-28T18:46:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 client.25694 shutdown 2024-03-28T18:46:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 msgr2=0x55ab38205900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff894005fd0 tx=0x7ff894004730 comp rx=0 tx=0).stop 2024-03-28T18:46:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 shutdown_connections 2024-03-28T18:46:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ab382675d0 0x55ab382699f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ab381fb480 0x55ab38205900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ab381f6310 0x55ab382048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.389+0000 7ff8ac62f740 1 --2- 172.21.15.145:0/2152878018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ab381f5920 0x55ab382043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.390+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 >> 172.21.15.145:0/2152878018 conn(0x55ab381f0e90 msgr2=0x55ab381ef790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.390+0000 7ff882ffd640 0 client.25694 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.390+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 shutdown_connections 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.390+0000 7ff8ac62f740 1 -- 172.21.15.145:0/2152878018 wait complete. 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.390+0000 7ff8ac62f740 20 client.25694 trim_cache size 0 max 16384 2024-03-28T18:46:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 msgr2=0x7ff88c0a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 0x7ff88c0a4fa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55ab381f6b70 tx=0x7ff8940098b0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 msgr2=0x7ff8a4115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff89800b450 tx=0x7ff89800b910 comp rx=0 tx=0).stop 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 shutdown_connections 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8a410d9c0 0x7ff8a41b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8a410b010 0x7ff8a4115210 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7ff88c0a2ae0 0x7ff88c0a4fa0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 --2- 172.21.15.145:0/3062515625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8a41086a0 0x7ff8a4114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.391+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 >> 172.21.15.145:0/3062515625 conn(0x7ff8a406e6e0 msgr2=0x7ff8a410d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.393+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 shutdown_connections 2024-03-28T18:46:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:05.393+0000 7ff8a9962640 1 -- 172.21.15.145:0/3062515625 wait complete. 2024-03-28T18:46:05.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:04 smithi192 ceph-mon[36589]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:46:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:04 smithi164 ceph-mon[36248]: pgmap v600: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 651 MiB/s wr, 1.25k op/s 2024-03-28T18:46:07.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:06 smithi145 ceph-mon[32151]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 1.54k op/s 2024-03-28T18:46:07.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:06 smithi192 ceph-mon[36589]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 1.54k op/s 2024-03-28T18:46:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:06 smithi164 ceph-mon[36248]: pgmap v601: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 1.54k op/s 2024-03-28T18:46:09.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:08 smithi145 ceph-mon[32151]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:46:09.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:09 smithi192 ceph-mon[36589]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:46:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:09 smithi164 ceph-mon[36248]: pgmap v602: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 649 MiB/s wr, 1.15k op/s 2024-03-28T18:46:11.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:11 smithi145 ceph-mon[32151]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 708 MiB/s wr, 1.21k op/s 2024-03-28T18:46:11.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:11 smithi192 ceph-mon[36589]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 708 MiB/s wr, 1.21k op/s 2024-03-28T18:46:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:11 smithi164 ceph-mon[36248]: pgmap v603: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 708 MiB/s wr, 1.21k op/s 2024-03-28T18:46:13.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:13 smithi145 ceph-mon[32151]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:13 smithi192 ceph-mon[36589]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:13 smithi164 ceph-mon[36248]: pgmap v604: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:15.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:15 smithi145 ceph-mon[32151]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 700 MiB/s wr, 1.34k op/s 2024-03-28T18:46:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:15 smithi192 ceph-mon[36589]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 700 MiB/s wr, 1.34k op/s 2024-03-28T18:46:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:15 smithi164 ceph-mon[36248]: pgmap v605: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 700 MiB/s wr, 1.34k op/s 2024-03-28T18:46:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:17 smithi192 ceph-mon[36589]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 758 MiB/s wr, 1.46k op/s 2024-03-28T18:46:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:17 smithi164 ceph-mon[36248]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 758 MiB/s wr, 1.46k op/s 2024-03-28T18:46:17.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:17 smithi145 ceph-mon[32151]: pgmap v606: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 758 MiB/s wr, 1.46k op/s 2024-03-28T18:46:18.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.663+0000 7f7ecec3f640 1 -- 172.21.15.145:0/678481223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec81098b0 msgr2=0x7f7ec8110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.663+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/678481223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec81098b0 0x7f7ec8110140 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009860 tx=0x7f7eb802bbb0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.670+0000 7f7ecec3f640 1 -- 172.21.15.145:0/678481223 shutdown_connections 2024-03-28T18:46:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.670+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/678481223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec81098b0 0x7f7ec8110140 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.670+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/678481223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.670+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/678481223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8107d00 0x7f7ec8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.670+0000 7f7ecec3f640 1 -- 172.21.15.145:0/678481223 >> 172.21.15.145:0/678481223 conn(0x7f7ec8076cc0 msgr2=0x7f7ec80790e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.675+0000 7f7ecec3f640 1 -- 172.21.15.145:0/678481223 shutdown_connections 2024-03-28T18:46:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.678+0000 7f7ecec3f640 1 -- 172.21.15.145:0/678481223 wait complete. 2024-03-28T18:46:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.679+0000 7f7ecec3f640 1 Processor -- start 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 -- start start 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8107d00 0x7f7ec8072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 0x7f7ec8071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec806bb10 con 0x7f7ec8107d00 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec806b990 con 0x7f7ec81098b0 2024-03-28T18:46:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.680+0000 7f7ecec3f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec806bc90 con 0x7f7ec8108ef0 2024-03-28T18:46:18.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.683+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 0x7f7ec8071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.683 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.683+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.685 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.685+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 0x7f7ec8071160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42286/0 (socket says 172.21.15.145:42286) 2024-03-28T18:46:18.685 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.685+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40998/0 (socket says 172.21.15.145:40998) 2024-03-28T18:46:18.685 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.685+0000 7f7ec7fff640 1 -- 172.21.15.145:0/3333199494 learned_addr learned my addr 172.21.15.145:0/3333199494 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8107d00 0x7f7ec8072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ec7fff640 1 -- 172.21.15.145:0/3333199494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 msgr2=0x7f7ec8071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ec7fff640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 0x7f7ec8071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ec7fff640 1 -- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8107d00 msgr2=0x7f7ec8072b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ec7fff640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8107d00 0x7f7ec8072b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.687+0000 7f7ec7fff640 1 -- 172.21.15.145:0/3333199494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8071810 con 0x7f7ec8108ef0 2024-03-28T18:46:18.694 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.692+0000 7f7ec7fff640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00d950 tx=0x7f7ebc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.696+0000 7f7ec5ffb640 1 -- 172.21.15.145:0/3333199494 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc017070 con 0x7f7ec8108ef0 2024-03-28T18:46:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.696+0000 7f7ec5ffb640 1 -- 172.21.15.145:0/3333199494 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc009c30 con 0x7f7ec8108ef0 2024-03-28T18:46:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.696+0000 7f7ec5ffb640 1 -- 172.21.15.145:0/3333199494 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00f6b0 con 0x7f7ec8108ef0 2024-03-28T18:46:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.697+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec8071b00 con 0x7f7ec8108ef0 2024-03-28T18:46:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.697+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec8079ef0 con 0x7f7ec8108ef0 2024-03-28T18:46:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.703+0000 7f7ed1820740 1 Processor -- start 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.703+0000 7f7ed1820740 1 -- start start 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b93bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b943910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b8a0b10 con 0x55ad4b93c310 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b849900 con 0x55ad4b941480 2024-03-28T18:46:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.704+0000 7f7ed1820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b89cbb0 con 0x55ad4b93b920 2024-03-28T18:46:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.705+0000 7f7ecc9db640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.705+0000 7f7ecc9db640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53488/0 (socket says 172.21.15.145:53488) 2024-03-28T18:46:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.705+0000 7f7ecc9db640 1 -- 172.21.15.145:0/1840916226 learned_addr learned my addr 172.21.15.145:0/1840916226 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecc9db640 1 -- 172.21.15.145:0/1840916226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 msgr2=0x55ad4b93bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.707 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b93bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b93bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecc9db640 1 -- 172.21.15.145:0/1840916226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 msgr2=0x55ad4b943910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b943910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.706+0000 7f7ecc9db640 1 -- 172.21.15.145:0/1840916226 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad4b5b5210 con 0x55ad4b93c310 2024-03-28T18:46:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.708+0000 7f7ec5ffb640 1 -- 172.21.15.145:0/3333199494 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7ebc00f850 con 0x7f7ec8108ef0 2024-03-28T18:46:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.711+0000 7f7ec5ffb640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 0x7f7eb00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.712 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.712+0000 7f7ec5ffb640 1 -- 172.21.15.145:0/3333199494 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f7ebc0d0b40 con 0x7f7ec8108ef0 2024-03-28T18:46:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.715+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009f50 tx=0x7f7ec00108a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1a575db8531f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.716+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 0x7f7eb00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.718+0000 7f7eaeffd640 1 -- 172.21.15.145:0/1840916226 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0004d20 con 0x55ad4b93c310 2024-03-28T18:46:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.718+0000 7f7eaeffd640 1 -- 172.21.15.145:0/1840916226 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ec000eb40 con 0x55ad4b93c310 2024-03-28T18:46:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.719+0000 7f7ed1820740 1 -- 172.21.15.145:0/1840916226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 msgr2=0x55ad4b940e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.719+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009f50 tx=0x7f7ec00108a0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.723+0000 7f7ed1820740 1 -- 172.21.15.145:0/1840916226 shutdown_connections 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.723+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b943910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.723+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b940e10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.723+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1840916226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b93bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.723+0000 7f7ed1820740 1 -- 172.21.15.145:0/1840916226 >> 172.21.15.145:0/1840916226 conn(0x55ad4b936e90 msgr2=0x55ad4b937290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.724+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 0x7f7eb00a4de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009830 tx=0x7f7eb8034b70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.731+0000 7f7ed1820740 1 -- 172.21.15.145:0/1840916226 shutdown_connections 2024-03-28T18:46:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.731+0000 7f7ed1820740 1 -- 172.21.15.145:0/1840916226 wait complete. 2024-03-28T18:46:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.732+0000 7f7ed1820740 1 -- start start 2024-03-28T18:46:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.732+0000 7f7ed1820740 1 Processor -- start 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b944480 con 0x55ad4b93b920 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b85cd50 con 0x55ad4b941480 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad4b944600 con 0x55ad4b93c310 2024-03-28T18:46:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.733+0000 7f7ed1820740 10 client.0 _ll_register_callbacks cb 0x55ad4b85c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.735+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.735+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.735+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:42300/0 (socket says 172.21.15.145:42300) 2024-03-28T18:46:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.735+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3847340114 learned_addr learned my addr 172.21.15.145:0/3847340114 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.737+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93c310 msgr2=0x55ad4b946220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.737+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.737+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3847340114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93b920 msgr2=0x55ad4b945ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.737+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.738 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.737+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3847340114 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad4b681620 con 0x55ad4b941480 2024-03-28T18:46:18.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.742+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8034be0 tx=0x7f7eb802bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9029554cd2d7b21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.744 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.744+0000 7f7ea7fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.744+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8045070 con 0x55ad4b941480 2024-03-28T18:46:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.744+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f7eb8005040 con 0x55ad4b941480 2024-03-28T18:46:18.745 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.744+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ec000eb40 con 0x55ad4b941480 2024-03-28T18:46:18.746 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.746+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7eb8039070 con 0x55ad4b941480 2024-03-28T18:46:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.752+0000 7f7ed1820740 10 client.25703 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:18.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.752+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ad4b9505d0 con 0x55ad4b941480 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.757+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ec000eb40 con 0x55ad4b941480 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7eb80035f0 con 0x55ad4b941480 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 10 client.25703 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 10 client.25703 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 20 client.25703 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b952100 0x55ad4b954560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 4 client.25703 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:18.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.758+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad4b8ae050 con 0x55ad4b952100 2024-03-28T18:46:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.774+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b952100 0x55ad4b954560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.791+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b952100 0x55ad4b954560 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49f31e8ac912d2b server_cookie=ff3fc4cb6a26dd48 in_seq=0 out_seq=0 2024-03-28T18:46:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.791+0000 7f7ea7fff640 10 client.25703 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.797+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3847340114 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad4b8ae050 con 0x55ad4b952100 2024-03-28T18:46:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.797+0000 7f7ea7fff640 10 client.25703 handle_command_reply: tid=0 2024-03-28T18:46:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.797+0000 7f7ed1820740 1 client.25703 shutdown 2024-03-28T18:46:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.797+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 msgr2=0x55ad4b94d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.798 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.797+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8034be0 tx=0x7f7eb802bbb0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 shutdown_connections 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b952100 0x55ad4b954560 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b941480 0x55ad4b94d1f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3847340114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 >> 172.21.15.145:0/3847340114 conn(0x55ad4b936e90 msgr2=0x55ad4b936020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ea7fff640 0 client.25703 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.801+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 shutdown_connections 2024-03-28T18:46:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.806+0000 7f7ed1820740 1 -- 172.21.15.145:0/3847340114 wait complete. 2024-03-28T18:46:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.806+0000 7f7ed1820740 20 client.25703 trim_cache size 0 max 16384 2024-03-28T18:46:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.851+0000 7f7ed1820740 1 Processor -- start 2024-03-28T18:46:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 -- start start 2024-03-28T18:46:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.852 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400fb80 con 0x55ad4b941480 2024-03-28T18:46:18.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4011500 con 0x55ad4b93c310 2024-03-28T18:46:18.853 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.852+0000 7f7ed1820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40101e0 con 0x55ad4b93b920 2024-03-28T18:46:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.854+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.854+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:53502/0 (socket says 172.21.15.145:53502) 2024-03-28T18:46:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.854+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/1695934940 learned_addr learned my addr 172.21.15.145:0/1695934940 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.855+0000 7f7ec7fff640 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/1695934940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 msgr2=0x55ad4b945ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/1695934940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93c310 msgr2=0x55ad4b946220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.856+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/1695934940 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad4b9408b0 con 0x55ad4b941480 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.857+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7eb802c0f0 tx=0x7f7eb8003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d8a615d5a0b44e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.857+0000 7f7eae7fc640 1 -- 172.21.15.145:0/1695934940 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8045070 con 0x55ad4b941480 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.857+0000 7f7eae7fc640 1 -- 172.21.15.145:0/1695934940 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7eb8051070 con 0x55ad4b941480 2024-03-28T18:46:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.857+0000 7f7eae7fc640 1 -- 172.21.15.145:0/1695934940 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb804cd30 con 0x55ad4b941480 2024-03-28T18:46:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.859+0000 7f7ed1820740 1 -- 172.21.15.145:0/1695934940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 msgr2=0x55ad4b94d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.859+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7eb802c0f0 tx=0x7f7eb8003fd0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.864+0000 7f7ed1820740 1 -- 172.21.15.145:0/1695934940 shutdown_connections 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.864+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b941480 0x55ad4b94d1b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.864+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93c310 0x55ad4b946220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.864+0000 7f7ed1820740 1 --2- 172.21.15.145:0/1695934940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b93b920 0x55ad4b945ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.864+0000 7f7ed1820740 1 -- 172.21.15.145:0/1695934940 >> 172.21.15.145:0/1695934940 conn(0x55ad4b936e90 msgr2=0x55ad4b937270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:18.866 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.866+0000 7f7ed1820740 1 -- 172.21.15.145:0/1695934940 shutdown_connections 2024-03-28T18:46:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.871+0000 7f7ed1820740 1 -- 172.21.15.145:0/1695934940 wait complete. 2024-03-28T18:46:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.871+0000 7f7ed1820740 1 -- start start 2024-03-28T18:46:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.871+0000 7f7ed1820740 1 Processor -- start 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93b920 0x55ad4b94a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b94ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4010ea0 con 0x55ad4b93c310 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4010840 con 0x55ad4b93b920 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400dd40 con 0x55ad4b941480 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.872+0000 7f7ed1820740 10 client.0 _ll_register_callbacks cb 0x7f7eb8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.873+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.873+0000 7f7ecd1dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:41044/0 (socket says 172.21.15.145:41044) 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.873+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3159543499 learned_addr learned my addr 172.21.15.145:0/3159543499 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.873+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93b920 0x55ad4b94a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.874+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3159543499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93b920 msgr2=0x55ad4b94a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.874+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93b920 0x55ad4b94a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.874+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3159543499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 msgr2=0x55ad4b94ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.874+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b94ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.874+0000 7f7ecd1dc640 1 -- 172.21.15.145:0/3159543499 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad4b94c250 con 0x55ad4b941480 2024-03-28T18:46:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.876+0000 7f7ecd1dc640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8005b60 tx=0x7f7eb8003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f709d1e7ef443499 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.877+0000 7f7ea7fff640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:46:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.877+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8045040 con 0x55ad4b941480 2024-03-28T18:46:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.877+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f7eb804cad0 con 0x55ad4b941480 2024-03-28T18:46:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.878+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7eb8051070 con 0x55ad4b941480 2024-03-28T18:46:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.880+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7eb8071050 con 0x55ad4b941480 2024-03-28T18:46:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.880+0000 7f7ed1820740 10 client.25792 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.880+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ad4b94d560 con 0x55ad4b941480 2024-03-28T18:46:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.880+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7eb8051070 con 0x55ad4b941480 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.880+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7eb8056450 con 0x55ad4b941480 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 10 client.25792 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 10 client.25792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 20 client.25792 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b9ae080 0x55ad4b9b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 4 client.25792 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:46:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.881+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ad4b8a83a0 con 0x55ad4b9ae080 2024-03-28T18:46:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.883+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b9ae080 0x55ad4b9b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.895+0000 7f7ecc9db640 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b9ae080 0x55ad4b9b04a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606a40323d939b3 server_cookie=6c115e08a0253f0d in_seq=0 out_seq=0 2024-03-28T18:46:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:18.895+0000 7f7ea7fff640 10 client.25792 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:19 smithi192 ceph-mon[36589]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 665 MiB/s wr, 1.21k op/s 2024-03-28T18:46:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:19 smithi164 ceph-mon[36248]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 665 MiB/s wr, 1.21k op/s 2024-03-28T18:46:19.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:19 smithi145 ceph-mon[32151]: pgmap v607: 129 pgs: 129 active+clean; 10 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 665 MiB/s wr, 1.21k op/s 2024-03-28T18:46:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.746+0000 7f7ea7fff640 1 -- 172.21.15.145:0/3159543499 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ad4b8a83a0 con 0x55ad4b9ae080 2024-03-28T18:46:20.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.746+0000 7f7ea7fff640 10 client.25792 handle_command_reply: tid=0 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.746+0000 7f7ed1820740 1 client.25792 shutdown 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 msgr2=0x55ad4b94bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8005b60 tx=0x7f7eb8003e30 comp rx=0 tx=0).stop 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 shutdown_connections 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ad4b9ae080 0x55ad4b9b04a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ad4b941480 0x55ad4b94bc60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ad4b93c310 0x55ad4b94ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 --2- 172.21.15.145:0/3159543499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad4b93b920 0x55ad4b94a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 >> 172.21.15.145:0/3159543499 conn(0x55ad4b936e90 msgr2=0x55ad4b945c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ea7fff640 0 client.25792 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 shutdown_connections 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 1 -- 172.21.15.145:0/3159543499 wait complete. 2024-03-28T18:46:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.747+0000 7f7ed1820740 20 client.25792 trim_cache size 0 max 16384 2024-03-28T18:46:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:46:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:46:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:46:20.749 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:46:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.750+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 msgr2=0x7f7eb00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.750+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 0x7f7eb00a4de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009830 tx=0x7f7eb8034b70 comp rx=0 tx=0).stop 2024-03-28T18:46:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.750+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 msgr2=0x7f7ec8073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:20.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.750+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00d950 tx=0x7f7ebc00de10 comp rx=0 tx=0).stop 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 shutdown_connections 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec81098b0 0x7f7ec8071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7eb00a2920 0x7f7eb00a4de0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ec8108ef0 0x7f7ec8073060 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 --2- 172.21.15.145:0/3333199494 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8107d00 0x7f7ec8072b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:20.752 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.751+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 >> 172.21.15.145:0/3333199494 conn(0x7f7ec8076cc0 msgr2=0x7f7ec810c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.753+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 shutdown_connections 2024-03-28T18:46:20.754 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:20.754+0000 7f7ecec3f640 1 -- 172.21.15.145:0/3333199494 wait complete. 2024-03-28T18:46:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:21 smithi192 ceph-mon[36589]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 727 MiB/s wr, 1.30k op/s 2024-03-28T18:46:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:21 smithi164 ceph-mon[36248]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 727 MiB/s wr, 1.30k op/s 2024-03-28T18:46:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:21 smithi145 ceph-mon[32151]: pgmap v608: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 727 MiB/s wr, 1.30k op/s 2024-03-28T18:46:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:23 smithi192 ceph-mon[36589]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.30k op/s 2024-03-28T18:46:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:23 smithi164 ceph-mon[36248]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.30k op/s 2024-03-28T18:46:23.532 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:23 smithi145 ceph-mon[32151]: pgmap v609: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 MiB/s wr, 1.30k op/s 2024-03-28T18:46:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:25 smithi192 ceph-mon[36589]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 702 MiB/s wr, 1.27k op/s 2024-03-28T18:46:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:25 smithi164 ceph-mon[36248]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 702 MiB/s wr, 1.27k op/s 2024-03-28T18:46:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:25 smithi145 ceph-mon[32151]: pgmap v610: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 702 MiB/s wr, 1.27k op/s 2024-03-28T18:46:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:27 smithi192 ceph-mon[36589]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 722 MiB/s wr, 1.35k op/s 2024-03-28T18:46:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:27 smithi164 ceph-mon[36248]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 722 MiB/s wr, 1.35k op/s 2024-03-28T18:46:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:27 smithi145 ceph-mon[32151]: pgmap v611: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 722 MiB/s wr, 1.35k op/s 2024-03-28T18:46:29.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:29 smithi192 ceph-mon[36589]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 712 MiB/s wr, 1.39k op/s 2024-03-28T18:46:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:29 smithi164 ceph-mon[36248]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 712 MiB/s wr, 1.39k op/s 2024-03-28T18:46:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:29 smithi145 ceph-mon[32151]: pgmap v612: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 712 MiB/s wr, 1.39k op/s 2024-03-28T18:46:31.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:31 smithi192 ceph-mon[36589]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.40k op/s 2024-03-28T18:46:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:31 smithi164 ceph-mon[36248]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.40k op/s 2024-03-28T18:46:31.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:31 smithi145 ceph-mon[32151]: pgmap v613: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 724 MiB/s wr, 1.40k op/s 2024-03-28T18:46:33.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:33 smithi192 ceph-mon[36589]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:33 smithi164 ceph-mon[36248]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:33.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:33 smithi145 ceph-mon[32151]: pgmap v614: 129 pgs: 129 active+clean; 10 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 MiB/s wr, 1.26k op/s 2024-03-28T18:46:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.980+0000 7f575ffff640 1 -- 172.21.15.145:0/607234173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010dbb0 msgr2=0x7f5760110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.980+0000 7f575ffff640 1 --2- 172.21.15.145:0/607234173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010dbb0 0x7f5760110070 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5758009900 tx=0x7f575802e020 comp rx=0 tx=0).stop 2024-03-28T18:46:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.981+0000 7f575ffff640 1 -- 172.21.15.145:0/607234173 shutdown_connections 2024-03-28T18:46:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.981+0000 7f575ffff640 1 --2- 172.21.15.145:0/607234173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010dbb0 0x7f5760110070 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.981+0000 7f575ffff640 1 --2- 172.21.15.145:0/607234173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010b0f0 0x7f576010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.981+0000 7f575ffff640 1 --2- 172.21.15.145:0/607234173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5760108730 0x7f576010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.981+0000 7f575ffff640 1 -- 172.21.15.145:0/607234173 >> 172.21.15.145:0/607234173 conn(0x7f576006e6d0 msgr2=0x7f5760070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.985+0000 7f575ffff640 1 -- 172.21.15.145:0/607234173 shutdown_connections 2024-03-28T18:46:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.985+0000 7f575ffff640 1 -- 172.21.15.145:0/607234173 wait complete. 2024-03-28T18:46:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.985+0000 7f575ffff640 1 Processor -- start 2024-03-28T18:46:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 -- start start 2024-03-28T18:46:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5760108730 0x7f57601a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010b0f0 0x7f57601a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5760113890 con 0x7f576010b0f0 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5760113710 con 0x7f576010dbb0 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575ffff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5760113a10 con 0x7f5760108730 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:49944/0 (socket says 172.21.15.145:49944) 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.986+0000 7f575f7fe640 1 -- 172.21.15.145:0/2489949174 learned_addr learned my addr 172.21.15.145:0/2489949174 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 -- 172.21.15.145:0/2489949174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5760108730 msgr2=0x7f57601a22e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5760108730 0x7f57601a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 -- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010b0f0 msgr2=0x7f57601a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010b0f0 0x7f57601a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 -- 172.21.15.145:0/2489949174 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57601a9f00 con 0x7f576010dbb0 2024-03-28T18:46:33.988 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.987+0000 7f575f7fe640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5758009850 tx=0x7f5758003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.993+0000 7f5747fff640 1 -- 172.21.15.145:0/2489949174 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758044070 con 0x7f576010dbb0 2024-03-28T18:46:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.994+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57601aa190 con 0x7f576010dbb0 2024-03-28T18:46:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.994+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57601aa700 con 0x7f576010dbb0 2024-03-28T18:46:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.994+0000 7f5747fff640 1 -- 172.21.15.145:0/2489949174 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5758004520 con 0x7f576010dbb0 2024-03-28T18:46:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.994+0000 7f5747fff640 1 -- 172.21.15.145:0/2489949174 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575803b030 con 0x7f576010dbb0 2024-03-28T18:46:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:33.997+0000 7f5747fff640 1 -- 172.21.15.145:0/2489949174 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5758047070 con 0x7f576010dbb0 2024-03-28T18:46:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.003+0000 7f576823c740 1 Processor -- start 2024-03-28T18:46:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.003+0000 7f576823c740 1 -- start start 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.003+0000 7f576823c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2beb20 0x55eb5e2bef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f576823c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f576823c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 0x55eb5e2c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f576823c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e223b10 con 0x55eb5e2bf510 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f576823c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e1cc900 con 0x55eb5e2beb20 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f576823c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e21fbb0 con 0x55eb5e2c4680 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f575effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f575effd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:42530/0 (socket says 172.21.15.145:42530) 2024-03-28T18:46:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.004+0000 7f575effd640 1 -- 172.21.15.145:0/2388513949 learned_addr learned my addr 172.21.15.145:0/2388513949 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f5747fff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 0x7f57480a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 0x55eb5e2c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2beb20 0x55eb5e2bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 -- 172.21.15.145:0/2388513949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2beb20 msgr2=0x55eb5e2bef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2beb20 0x55eb5e2bef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 -- 172.21.15.145:0/2388513949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 msgr2=0x55eb5e2c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.005+0000 7f575e7fc640 1 -- 172.21.15.145:0/2388513949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb5df38210 con 0x55eb5e2c4680 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.006+0000 7f575e7fc640 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 0x55eb5e2c6b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57601a37f0 tx=0x7f575400b470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d69b7141946bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.006+0000 7f57457fa640 1 -- 172.21.15.145:0/2388513949 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754004870 con 0x55eb5e2c4680 2024-03-28T18:46:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.006+0000 7f575f7fe640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 0x7f57480a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.009+0000 7f57457fa640 1 -- 172.21.15.145:0/2388513949 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57540060d0 con 0x55eb5e2c4680 2024-03-28T18:46:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.009+0000 7f57457fa640 1 -- 172.21.15.145:0/2388513949 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575401a7a0 con 0x55eb5e2c4680 2024-03-28T18:46:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.013+0000 7f5747fff640 1 -- 172.21.15.145:0/2489949174 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f57580f0090 con 0x7f576010dbb0 2024-03-28T18:46:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f575f7fe640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 0x7f57480a4e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f57580b8a90 tx=0x7f57580b8200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 -- 172.21.15.145:0/2388513949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 msgr2=0x55eb5e2c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 0x55eb5e2c6b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57601a37f0 tx=0x7f575400b470 comp rx=0 tx=0).stop 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 -- 172.21.15.145:0/2388513949 shutdown_connections 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2c4680 0x55eb5e2c6b10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 --2- 172.21.15.145:0/2388513949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2beb20 0x55eb5e2bef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.014+0000 7f576823c740 1 -- 172.21.15.145:0/2388513949 >> 172.21.15.145:0/2388513949 conn(0x55eb5e2ba090 msgr2=0x55eb5e2ba490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.017+0000 7f576823c740 1 -- 172.21.15.145:0/2388513949 shutdown_connections 2024-03-28T18:46:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.018+0000 7f576823c740 1 -- 172.21.15.145:0/2388513949 wait complete. 2024-03-28T18:46:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.018+0000 7f576823c740 1 -- start start 2024-03-28T18:46:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.018+0000 7f576823c740 1 Processor -- start 2024-03-28T18:46:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e2c75d0 con 0x55eb5e2bf510 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e1dfd50 con 0x55eb5e2c4680 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb5e2c7750 con 0x55eb5e2beb20 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.021+0000 7f576823c740 10 client.0 _ll_register_callbacks cb 0x55eb5e1df720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40288/0 (socket says 172.21.15.145:40288) 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 -- 172.21.15.145:0/3104022132 learned_addr learned my addr 172.21.15.145:0/3104022132 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575effd640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 -- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 msgr2=0x55eb5e2d1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 -- 172.21.15.145:0/3104022132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 msgr2=0x55eb5e2d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.022+0000 7f575e7fc640 1 -- 172.21.15.145:0/3104022132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb5e004620 con 0x55eb5e2c4680 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.023+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d8b90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f57540120b0 tx=0x7f5754004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e006d1293f0428c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.023+0000 7f5742ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754004aa0 con 0x55eb5e2c4680 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57540060d0 con 0x55eb5e2c4680 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5754022040 con 0x55eb5e2c4680 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5754004c40 con 0x55eb5e2c4680 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f576823c740 10 client.25718 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.024+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eb5e2cb900 con 0x55eb5e2c4680 2024-03-28T18:46:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57540060d0 con 0x55eb5e2c4680 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f575400c520 con 0x55eb5e2c4680 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 10 client.25718 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 20 client.25718 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e2cd270 0x55eb5e2cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 4 client.25718 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.028+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb5e231050 con 0x55eb5e2cd270 2024-03-28T18:46:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.029+0000 7f575effd640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e2cd270 0x55eb5e2cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.032+0000 7f575effd640 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e2cd270 0x55eb5e2cf690 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acefa5469edb9875 server_cookie=316d349dab7e8406 in_seq=0 out_seq=0 2024-03-28T18:46:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.033+0000 7f5742ff5640 10 client.25718 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.034+0000 7f5742ff5640 1 -- 172.21.15.145:0/3104022132 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eb5e231050 con 0x55eb5e2cd270 2024-03-28T18:46:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.034+0000 7f5742ff5640 10 client.25718 handle_command_reply: tid=0 2024-03-28T18:46:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.034+0000 7f576823c740 1 client.25718 shutdown 2024-03-28T18:46:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.038+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 msgr2=0x55eb5e2d8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.038+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d8b90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f57540120b0 tx=0x7f5754004770 comp rx=0 tx=0).stop 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 shutdown_connections 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e2cd270 0x55eb5e2cf690 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d8b90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 --2- 172.21.15.145:0/3104022132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 >> 172.21.15.145:0/3104022132 conn(0x55eb5e2ba090 msgr2=0x55eb5e2b9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:34.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.042+0000 7f5742ff5640 0 client.25718 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.045+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 shutdown_connections 2024-03-28T18:46:34.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.045+0000 7f576823c740 1 -- 172.21.15.145:0/3104022132 wait complete. 2024-03-28T18:46:34.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.046+0000 7f576823c740 20 client.25718 trim_cache size 0 max 16384 2024-03-28T18:46:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.079+0000 7f576823c740 1 Processor -- start 2024-03-28T18:46:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.079+0000 7f576823c740 1 -- start start 2024-03-28T18:46:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f576823c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f576823c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:40304/0 (socket says 172.21.15.145:40304) 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f575e7fc640 1 -- 172.21.15.145:0/3519603935 learned_addr learned my addr 172.21.15.145:0/3519603935 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f576823c740 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.080+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c00fb30 con 0x55eb5e2c4680 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575effd640 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0114b0 con 0x55eb5e2bf510 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c010190 con 0x55eb5e2beb20 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575e7fc640 1 -- 172.21.15.145:0/3519603935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2bf510 msgr2=0x55eb5e2d1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575e7fc640 1 -- 172.21.15.145:0/3519603935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2c4680 msgr2=0x55eb5e2d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.081+0000 7f575e7fc640 1 -- 172.21.15.145:0/3519603935 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb5e2c3a80 con 0x55eb5e2beb20 2024-03-28T18:46:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.082+0000 7f575e7fc640 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f575400c8f0 tx=0x7f575400d5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19cc20dde9d37649 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.083+0000 7f5744ff9640 1 -- 172.21.15.145:0/3519603935 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754004a80 con 0x55eb5e2beb20 2024-03-28T18:46:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.083+0000 7f5744ff9640 1 -- 172.21.15.145:0/3519603935 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5754011040 con 0x55eb5e2beb20 2024-03-28T18:46:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.083+0000 7f5744ff9640 1 -- 172.21.15.145:0/3519603935 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754007b80 con 0x55eb5e2beb20 2024-03-28T18:46:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 msgr2=0x55eb5e2ba470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f575400c8f0 tx=0x7f575400d5c0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 shutdown_connections 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 --2- 172.21.15.145:0/3519603935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2beb20 0x55eb5e2ba470 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.085+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 >> 172.21.15.145:0/3519603935 conn(0x55eb5e2ba090 msgr2=0x55eb5e2c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.086+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 shutdown_connections 2024-03-28T18:46:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.087+0000 7f576823c740 1 -- 172.21.15.145:0/3519603935 wait complete. 2024-03-28T18:46:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.087+0000 7f576823c740 1 -- start start 2024-03-28T18:46:34.097 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.088+0000 7f576823c740 1 Processor -- start 2024-03-28T18:46:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.098+0000 7f576823c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.098+0000 7f576823c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.099+0000 7f576823c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.099+0000 7f576823c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c010e50 con 0x55eb5e2beb20 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.099+0000 7f576823c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0107f0 con 0x55eb5e2c4680 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.099+0000 7f576823c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c00dcf0 con 0x55eb5e2bf510 2024-03-28T18:46:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.099+0000 7f576823c740 10 client.0 _ll_register_callbacks cb 0x7f5754006620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.100+0000 7f575effd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.100+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.100+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.100+0000 7f575e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:42558/0 (socket says 172.21.15.145:42558) 2024-03-28T18:46:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.100+0000 7f575e7fc640 1 -- 172.21.15.145:0/32952137 learned_addr learned my addr 172.21.15.145:0/32952137 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.101+0000 7f575effd640 1 -- 172.21.15.145:0/32952137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 msgr2=0x55eb5e2d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.101+0000 7f575effd640 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.101+0000 7f575effd640 1 -- 172.21.15.145:0/32952137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 msgr2=0x55eb5e2d5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.101+0000 7f575effd640 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.101+0000 7f575effd640 1 -- 172.21.15.145:0/32952137 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb5e2d71e0 con 0x55eb5e2bf510 2024-03-28T18:46:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.104+0000 7f575effd640 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d5c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55eb5e2cff10 tx=0x7f5750009110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a44201d7c2088478 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.105+0000 7f5742ff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:46:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.110+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750028060 con 0x55eb5e2bf510 2024-03-28T18:46:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.111+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5750023460 con 0x55eb5e2bf510 2024-03-28T18:46:34.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.115+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5754011040 con 0x55eb5e2bf510 2024-03-28T18:46:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.120+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f575000f040 con 0x55eb5e2bf510 2024-03-28T18:46:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.122+0000 7f576823c740 10 client.25804 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:34.123 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.122+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eb5e3302a0 con 0x55eb5e2bf510 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5750023b40 con 0x55eb5e2bf510 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f57500168e0 con 0x55eb5e2bf510 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 10 client.25804 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 10 client.25804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 20 client.25804 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:34.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e330bb0 0x55eb5e332fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 4 client.25804 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:46:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.130+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f5754011040 con 0x55eb5e330bb0 2024-03-28T18:46:34.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.131+0000 7f575e7fc640 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e330bb0 0x55eb5e332fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:34.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.133+0000 7f575e7fc640 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e330bb0 0x55eb5e332fd0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3c3e89876d3f8b server_cookie=f7926ca9fdf23739 in_seq=0 out_seq=0 2024-03-28T18:46:34.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:34.133+0000 7f5742ff5640 10 client.25804 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:35.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:35 smithi192 ceph-mon[36589]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 707 MiB/s wr, 1.32k op/s 2024-03-28T18:46:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:35 smithi164 ceph-mon[36248]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 707 MiB/s wr, 1.32k op/s 2024-03-28T18:46:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:35 smithi145 ceph-mon[32151]: pgmap v615: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 707 MiB/s wr, 1.32k op/s 2024-03-28T18:46:37.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:37 smithi192 ceph-mon[36589]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 728 MiB/s wr, 1.52k op/s 2024-03-28T18:46:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:37 smithi164 ceph-mon[36248]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 728 MiB/s wr, 1.52k op/s 2024-03-28T18:46:37.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:37 smithi145 ceph-mon[32151]: pgmap v616: 129 pgs: 129 active+clean; 10 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 728 MiB/s wr, 1.52k op/s 2024-03-28T18:46:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:39 smithi192 ceph-mon[36589]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 691 MiB/s wr, 1.54k op/s 2024-03-28T18:46:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f5742ff5640 1 -- 172.21.15.145:0/32952137 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f5754011040 con 0x55eb5e330bb0 2024-03-28T18:46:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f5742ff5640 10 client.25804 handle_command_reply: tid=0 2024-03-28T18:46:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 client.25804 shutdown 2024-03-28T18:46:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 msgr2=0x55eb5e2d5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d5c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55eb5e2cff10 tx=0x7f5750009110 comp rx=0 tx=0).stop 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 shutdown_connections 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55eb5e330bb0 0x55eb5e332fd0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb5e2c4680 0x55eb5e2d6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb5e2bf510 0x55eb5e2d5c50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 --2- 172.21.15.145:0/32952137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55eb5e2beb20 0x55eb5e2d5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 >> 172.21.15.145:0/32952137 conn(0x55eb5e2ba090 msgr2=0x55eb5e2cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.423+0000 7f5742ff5640 0 client.25804 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:39.426 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.424+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 shutdown_connections 2024-03-28T18:46:39.426 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.424+0000 7f576823c740 1 -- 172.21.15.145:0/32952137 wait complete. 2024-03-28T18:46:39.426 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.424+0000 7f576823c740 20 client.25804 trim_cache size 0 max 16384 2024-03-28T18:46:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:46:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:46:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:46:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:46:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.428+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 msgr2=0x7f57480a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.428+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 0x7f57480a4e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f57580b8a90 tx=0x7f57580b8200 comp rx=0 tx=0).stop 2024-03-28T18:46:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.428+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 msgr2=0x7f57601a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.428+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5758009850 tx=0x7f5758003ba0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.429+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 shutdown_connections 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.429+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f576010dbb0 0x7f57601a9890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.430+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f576010b0f0 0x7f57601a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.430+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f57480a2940 0x7f57480a4e00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.430+0000 7f575ffff640 1 --2- 172.21.15.145:0/2489949174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5760108730 0x7f57601a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.430+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 >> 172.21.15.145:0/2489949174 conn(0x7f576006e6d0 msgr2=0x7f576010ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.432+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 shutdown_connections 2024-03-28T18:46:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:39.432+0000 7f575ffff640 1 -- 172.21.15.145:0/2489949174 wait complete. 2024-03-28T18:46:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:39 smithi164 ceph-mon[36248]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 691 MiB/s wr, 1.54k op/s 2024-03-28T18:46:39.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:39 smithi145 ceph-mon[32151]: pgmap v617: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 691 MiB/s wr, 1.54k op/s 2024-03-28T18:46:41.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:41 smithi192 ceph-mon[36589]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.51k op/s 2024-03-28T18:46:41.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:41 smithi164 ceph-mon[36248]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.51k op/s 2024-03-28T18:46:41.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:41 smithi145 ceph-mon[32151]: pgmap v618: 129 pgs: 129 active+clean; 10 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.51k op/s 2024-03-28T18:46:43.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:43 smithi192 ceph-mon[36589]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 651 MiB/s wr, 1.38k op/s 2024-03-28T18:46:43.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:43 smithi164 ceph-mon[36248]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 651 MiB/s wr, 1.38k op/s 2024-03-28T18:46:43.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:43 smithi145 ceph-mon[32151]: pgmap v619: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 651 MiB/s wr, 1.38k op/s 2024-03-28T18:46:45.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:45 smithi164 ceph-mon[36248]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 MiB/s wr, 1.37k op/s 2024-03-28T18:46:45.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:45 smithi145 ceph-mon[32151]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 MiB/s wr, 1.37k op/s 2024-03-28T18:46:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:45 smithi192 ceph-mon[36589]: pgmap v620: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 MiB/s wr, 1.37k op/s 2024-03-28T18:46:47.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:47 smithi164 ceph-mon[36248]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 691 MiB/s wr, 1.37k op/s 2024-03-28T18:46:47.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:47 smithi145 ceph-mon[32151]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 691 MiB/s wr, 1.37k op/s 2024-03-28T18:46:47.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:47 smithi192 ceph-mon[36589]: pgmap v621: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 691 MiB/s wr, 1.37k op/s 2024-03-28T18:46:49.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:49 smithi164 ceph-mon[36248]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 678 MiB/s wr, 1.24k op/s 2024-03-28T18:46:49.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:49 smithi145 ceph-mon[32151]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 678 MiB/s wr, 1.24k op/s 2024-03-28T18:46:49.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:49 smithi192 ceph-mon[36589]: pgmap v622: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 678 MiB/s wr, 1.24k op/s 2024-03-28T18:46:51.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:51 smithi145 ceph-mon[32151]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.20k op/s 2024-03-28T18:46:51.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:51 smithi192 ceph-mon[36589]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.20k op/s 2024-03-28T18:46:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:51 smithi164 ceph-mon[36248]: pgmap v623: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.20k op/s 2024-03-28T18:46:52.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:52 smithi192 ceph-mon[36589]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 679 MiB/s wr, 1.19k op/s 2024-03-28T18:46:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:52 smithi164 ceph-mon[36248]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 679 MiB/s wr, 1.19k op/s 2024-03-28T18:46:52.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:52 smithi145 ceph-mon[32151]: pgmap v624: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 679 MiB/s wr, 1.19k op/s 2024-03-28T18:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.030+0000 7f9239947640 1 -- 172.21.15.145:0/552891521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 msgr2=0x7f922c0a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:53.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.030+0000 7f9239947640 1 --2- 172.21.15.145:0/552891521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c0a47b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9220009900 tx=0x7f922002bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.032+0000 7f9239947640 1 -- 172.21.15.145:0/552891521 shutdown_connections 2024-03-28T18:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.032+0000 7f9239947640 1 --2- 172.21.15.145:0/552891521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c0a47b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.032+0000 7f9239947640 1 --2- 172.21.15.145:0/552891521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f922c09d560 0x7f922c09d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.032 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.032+0000 7f9239947640 1 --2- 172.21.15.145:0/552891521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09c370 0x7f922c09c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.032+0000 7f9239947640 1 -- 172.21.15.145:0/552891521 >> 172.21.15.145:0/552891521 conn(0x7f922c00b4f0 msgr2=0x7f922c00d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:53.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.036+0000 7f9239947640 1 -- 172.21.15.145:0/552891521 shutdown_connections 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.041+0000 7f9239947640 1 -- 172.21.15.145:0/552891521 wait complete. 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.041+0000 7f9239947640 1 Processor -- start 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 -- start start 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f922c09c370 0x7f922c143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09d560 0x7f922c143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0a8020 con 0x7f922c09d560 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0a7ea0 con 0x7f922c09c370 2024-03-28T18:46:53.044 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.042+0000 7f9239947640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0a81a0 con 0x7f922c09df20 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50958/0 (socket says 172.21.15.145:50958) 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 -- 172.21.15.145:0/2760874710 learned_addr learned my addr 172.21.15.145:0/2760874710 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9233fff640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09d560 0x7f922c143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 -- 172.21.15.145:0/2760874710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f922c09c370 msgr2=0x7f922c143640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f922c09c370 0x7f922c143640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 -- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09d560 msgr2=0x7f922c143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:53.046 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09d560 0x7f922c143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.044+0000 7f9239146640 1 -- 172.21.15.145:0/2760874710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c133dd0 con 0x7f922c09df20 2024-03-28T18:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.045+0000 7f9239146640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9220036040 tx=0x7f9220004ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.047+0000 7f9231ffb640 1 -- 172.21.15.145:0/2760874710 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220043020 con 0x7f922c09df20 2024-03-28T18:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.047+0000 7f9231ffb640 1 -- 172.21.15.145:0/2760874710 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92200054e0 con 0x7f922c09df20 2024-03-28T18:46:53.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.047+0000 7f9231ffb640 1 -- 172.21.15.145:0/2760874710 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220003cf0 con 0x7f922c09df20 2024-03-28T18:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.048+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c134060 con 0x7f922c09df20 2024-03-28T18:46:53.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.048+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c134530 con 0x7f922c09df20 2024-03-28T18:46:53.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.056+0000 7f9231ffb640 1 -- 172.21.15.145:0/2760874710 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9220046020 con 0x7f922c09df20 2024-03-28T18:46:53.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.057+0000 7f9231ffb640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 0x7f92240a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.057+0000 7f9231ffb640 1 -- 172.21.15.145:0/2760874710 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f92200f10b0 con 0x7f922c09df20 2024-03-28T18:46:53.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 Processor -- start 2024-03-28T18:46:53.060 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 -- start start 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f337f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f33d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f28bb10 con 0x55ff9f337b30 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f234900 con 0x55ff9f338520 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.059+0000 7f923d78f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f287bb0 con 0x55ff9f33d690 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.060+0000 7f9239146640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.060+0000 7f9239146640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:50968/0 (socket says 172.21.15.145:50968) 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.060+0000 7f9239146640 1 -- 172.21.15.145:0/649535020 learned_addr learned my addr 172.21.15.145:0/649535020 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.060+0000 7f9238945640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 0x7f92240a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.060+0000 7f9238945640 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f337f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.062 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 -- 172.21.15.145:0/649535020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 msgr2=0x55ff9f33d020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f33d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 -- 172.21.15.145:0/649535020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 msgr2=0x55ff9f337f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f337f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 -- 172.21.15.145:0/649535020 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff9efa0210 con 0x55ff9f33d690 2024-03-28T18:46:53.063 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9238945640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 0x7f92240a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9228005d00 tx=0x7f922800a9d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.061+0000 7f9239146640 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92200b9aa0 tx=0x7f922002c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aa67ffd87a066d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.072+0000 7f921effd640 1 -- 172.21.15.145:0/649535020 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220004310 con 0x55ff9f33d690 2024-03-28T18:46:53.073 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.072+0000 7f921effd640 1 -- 172.21.15.145:0/649535020 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9220004c80 con 0x55ff9f33d690 2024-03-28T18:46:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.073+0000 7f923d78f740 1 -- 172.21.15.145:0/649535020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 msgr2=0x55ff9f33fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.073+0000 7f923d78f740 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92200b9aa0 tx=0x7f922002c6d0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.081+0000 7f923d78f740 1 -- 172.21.15.145:0/649535020 shutdown_connections 2024-03-28T18:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.081+0000 7f923d78f740 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f33fb20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.081+0000 7f923d78f740 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f33d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.081+0000 7f923d78f740 1 --2- 172.21.15.145:0/649535020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f337f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.083 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.081+0000 7f923d78f740 1 -- 172.21.15.145:0/649535020 >> 172.21.15.145:0/649535020 conn(0x55ff9f322090 msgr2=0x55ff9f322490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:53.094 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.092+0000 7f923d78f740 1 -- 172.21.15.145:0/649535020 shutdown_connections 2024-03-28T18:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.092+0000 7f923d78f740 1 -- 172.21.15.145:0/649535020 wait complete. 2024-03-28T18:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.092+0000 7f923d78f740 1 -- start start 2024-03-28T18:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.092+0000 7f923d78f740 1 Processor -- start 2024-03-28T18:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f341b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f338520 0x55ff9f3420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f340400 con 0x55ff9f337b30 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f247d50 con 0x55ff9f33d690 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9f340580 con 0x55ff9f338520 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.093+0000 7f923d78f740 10 client.0 _ll_register_callbacks cb 0x55ff9f247720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.094+0000 7f9239146640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.094+0000 7f9239146640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57268/0 (socket says 172.21.15.145:57268) 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.094+0000 7f9239146640 1 -- 172.21.15.145:0/3238414218 learned_addr learned my addr 172.21.15.145:0/3238414218 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:53.099 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.094+0000 7f9233fff640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f338520 0x55ff9f3420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.095+0000 7f9239146640 1 -- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f338520 msgr2=0x55ff9f3420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.095+0000 7f9239146640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f338520 0x55ff9f3420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.095+0000 7f9239146640 1 -- 172.21.15.145:0/3238414218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 msgr2=0x55ff9f341b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.095+0000 7f9239146640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f341b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.095+0000 7f9239146640 1 -- 172.21.15.145:0/3238414218 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff9f06c620 con 0x55ff9f33d690 2024-03-28T18:46:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.096+0000 7f9239146640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92200bd340 tx=0x7f9220003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba606c823768f197 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.109+0000 7f9213fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:53.113 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.109+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200da070 con 0x55ff9f33d690 2024-03-28T18:46:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.109+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f92200bdab0 con 0x55ff9f33d690 2024-03-28T18:46:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.111+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9220004c80 con 0x55ff9f33d690 2024-03-28T18:46:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.116+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92200bdd40 con 0x55ff9f33d690 2024-03-28T18:46:53.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.116+0000 7f923d78f740 10 client.25727 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:53.118 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.116+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ff9f340400 con 0x55ff9f33d690 2024-03-28T18:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.123+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9220004c80 con 0x55ff9f33d690 2024-03-28T18:46:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.123+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f92200bca90 con 0x55ff9f33d690 2024-03-28T18:46:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 10 client.25727 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 10 client.25727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 20 client.25727 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3294b0 0x55ff9f32b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 4 client.25727 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.125+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff9f299050 con 0x55ff9f3294b0 2024-03-28T18:46:53.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.132+0000 7f9238945640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3294b0 0x55ff9f32b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.133+0000 7f9238945640 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3294b0 0x55ff9f32b8f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43efe30ee48959f9 server_cookie=f8a99de8acd9b18e in_seq=0 out_seq=0 2024-03-28T18:46:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.136+0000 7f9213fff640 10 client.25727 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:53.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.137+0000 7f9213fff640 1 -- 172.21.15.145:0/3238414218 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ff9f299050 con 0x55ff9f3294b0 2024-03-28T18:46:53.138 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.137+0000 7f9213fff640 10 client.25727 handle_command_reply: tid=0 2024-03-28T18:46:53.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.141+0000 7f923d78f740 1 client.25727 shutdown 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 msgr2=0x55ff9f349090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92200bd340 tx=0x7f9220003700 comp rx=0 tx=0).stop 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 shutdown_connections 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3294b0 0x55ff9f32b8f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f33d690 0x55ff9f349090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f338520 0x55ff9f3420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 --2- 172.21.15.145:0/3238414218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f341b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.149 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.149+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 >> 172.21.15.145:0/3238414218 conn(0x55ff9f322090 msgr2=0x55ff9f321290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.150+0000 7f9213fff640 0 client.25727 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:53.150 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.150+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 shutdown_connections 2024-03-28T18:46:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.151+0000 7f923d78f740 1 -- 172.21.15.145:0/3238414218 wait complete. 2024-03-28T18:46:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.151+0000 7f923d78f740 20 client.25727 trim_cache size 0 max 16384 2024-03-28T18:46:53.190 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.190+0000 7f923d78f740 1 Processor -- start 2024-03-28T18:46:53.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.190+0000 7f923d78f740 1 -- start start 2024-03-28T18:46:53.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.190+0000 7f923d78f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f3252f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f923d78f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f923d78f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f325830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f923d78f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921800fb50 con 0x55ff9f337b30 2024-03-28T18:46:53.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f923d78f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92180114d0 con 0x55ff9f338520 2024-03-28T18:46:53.192 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f923d78f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92180101b0 con 0x55ff9f33d690 2024-03-28T18:46:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.191+0000 7f9233fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.192+0000 7f9233fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57280/0 (socket says 172.21.15.145:57280) 2024-03-28T18:46:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.192+0000 7f9233fff640 1 -- 172.21.15.145:0/2751373299 learned_addr learned my addr 172.21.15.145:0/2751373299 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.193+0000 7f9238945640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f3252f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.193+0000 7f9239146640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f325830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 -- 172.21.15.145:0/2751373299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 msgr2=0x55ff9f325830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.196 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f325830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 -- 172.21.15.145:0/2751373299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 msgr2=0x55ff9f3252f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f3252f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 -- 172.21.15.145:0/2751373299 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff9f342940 con 0x55ff9f338520 2024-03-28T18:46:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.195+0000 7f9233fff640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9234050950 tx=0x7f9234050480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bddfc728e59c5cce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.197 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.196+0000 7f9239146640 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f325830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.206+0000 7f921e7fc640 1 -- 172.21.15.145:0/2751373299 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234073dc0 con 0x55ff9f338520 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.206+0000 7f921e7fc640 1 -- 172.21.15.145:0/2751373299 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9234078070 con 0x55ff9f338520 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.207+0000 7f923d78f740 1 -- 172.21.15.145:0/2751373299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 msgr2=0x55ff9f349050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.207+0000 7f923d78f740 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9234050950 tx=0x7f9234050480 comp rx=0 tx=0).stop 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.208+0000 7f923d78f740 1 -- 172.21.15.145:0/2751373299 shutdown_connections 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.208+0000 7f923d78f740 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f325830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.208+0000 7f923d78f740 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f338520 0x55ff9f349050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.208 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.208+0000 7f923d78f740 1 --2- 172.21.15.145:0/2751373299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f337b30 0x55ff9f3252f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.209 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.208+0000 7f923d78f740 1 -- 172.21.15.145:0/2751373299 >> 172.21.15.145:0/2751373299 conn(0x55ff9f322090 msgr2=0x55ff9f3397a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.211+0000 7f923d78f740 1 -- 172.21.15.145:0/2751373299 shutdown_connections 2024-03-28T18:46:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.211+0000 7f923d78f740 1 -- 172.21.15.145:0/2751373299 wait complete. 2024-03-28T18:46:53.211 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.211+0000 7f923d78f740 1 -- start start 2024-03-28T18:46:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.211+0000 7f923d78f740 1 Processor -- start 2024-03-28T18:46:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f338520 0x55ff9f3463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f3b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9218010e70 con 0x55ff9f338520 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9218010810 con 0x55ff9f337b30 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921800dd10 con 0x55ff9f33d690 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f923d78f740 10 client.0 _ll_register_callbacks cb 0x7f92200bd060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:53.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.212+0000 7f9238945640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:57288/0 (socket says 172.21.15.145:57288) 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 -- 172.21.15.145:0/3775379225 learned_addr learned my addr 172.21.15.145:0/3775379225 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9239146640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f3b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 -- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 msgr2=0x55ff9f3b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f3b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.215 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 -- 172.21.15.145:0/3775379225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f338520 msgr2=0x55ff9f3463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f338520 0x55ff9f3463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.213+0000 7f9238945640 1 -- 172.21.15.145:0/3775379225 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff9f3b92b0 con 0x55ff9f337b30 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f9238945640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f922c144b90 tx=0x7f9228019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5045a0b0d55c5f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f9213fff640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92280044d0 con 0x55ff9f337b30 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f922800fde0 con 0x55ff9f337b30 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff9f2933a0 con 0x55ff9f337b30 2024-03-28T18:46:53.216 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.214+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9228035050 con 0x55ff9f337b30 2024-03-28T18:46:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.215+0000 7f923d78f740 10 client.25733 fetch_fsmap learned FSMap version 31 2024-03-28T18:46:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.215+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ff9f3ba530 con 0x55ff9f337b30 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.218+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ff9f2933a0 con 0x55ff9f337b30 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.218+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f922801e040 con 0x55ff9f337b30 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 10 client.25733 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 20 client.25733 populate_metadata read hostname 'smithi145' 2024-03-28T18:46:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3bae10 0x55ff9f3bd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 4 client.25733 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:46:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.220+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ff9f211710 con 0x55ff9f3bae10 2024-03-28T18:46:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.226+0000 7f9233fff640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3bae10 0x55ff9f3bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.228+0000 7f9233fff640 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3bae10 0x55ff9f3bd230 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d255cd88b520cd1 server_cookie=4f44a5e58abdbcbf in_seq=0 out_seq=0 2024-03-28T18:46:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:53.228+0000 7f9213fff640 10 client.25733 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:55.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:54 smithi145 ceph-mon[32151]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.25k op/s 2024-03-28T18:46:55.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:54 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:54 smithi192 ceph-mon[36589]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.25k op/s 2024-03-28T18:46:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:54 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:55.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:54 smithi164 ceph-mon[36248]: pgmap v625: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 692 MiB/s wr, 1.25k op/s 2024-03-28T18:46:55.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:54 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.801+0000 7f9213fff640 1 -- 172.21.15.145:0/3775379225 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ff9f211710 con 0x55ff9f3bae10 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f9213fff640 10 client.25733 handle_command_reply: tid=0 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 client.25733 shutdown 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 msgr2=0x55ff9f345e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f922c144b90 tx=0x7f9228019000 comp rx=0 tx=0).stop 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 shutdown_connections 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55ff9f3bae10 0x55ff9f3bd230 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.803 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff9f33d690 0x55ff9f3b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ff9f338520 0x55ff9f3463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 --2- 172.21.15.145:0/3775379225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ff9f337b30 0x55ff9f345e90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 >> 172.21.15.145:0/3775379225 conn(0x55ff9f322090 msgr2=0x55ff9f341b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.802+0000 7f9213fff640 0 client.25733 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.803+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 shutdown_connections 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.803+0000 7f923d78f740 1 -- 172.21.15.145:0/3775379225 wait complete. 2024-03-28T18:46:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.803+0000 7f923d78f740 20 client.25733 trim_cache size 0 max 16384 2024-03-28T18:46:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:46:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:46:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:46:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:46:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 msgr2=0x7f92240a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 0x7f92240a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9228005d00 tx=0x7f922800a9d0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 msgr2=0x7f922c133730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9220036040 tx=0x7f9220004ec0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 shutdown_connections 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f92240a2a00 0x7f92240a4ec0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f922c09df20 0x7f922c133730 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f922c09d560 0x7f922c143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 --2- 172.21.15.145:0/2760874710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f922c09c370 0x7f922c143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 >> 172.21.15.145:0/2760874710 conn(0x7f922c00b4f0 msgr2=0x7f922c0a0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 shutdown_connections 2024-03-28T18:46:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:46:56.804+0000 7f9239947640 1 -- 172.21.15.145:0/2760874710 wait complete. 2024-03-28T18:46:57.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:56 smithi145 ceph-mon[32151]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 740 MiB/s wr, 1.37k op/s 2024-03-28T18:46:57.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:57.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:57.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:56 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:46:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:56 smithi192 ceph-mon[36589]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 740 MiB/s wr, 1.37k op/s 2024-03-28T18:46:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:56 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:46:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:56 smithi164 ceph-mon[36248]: pgmap v626: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 740 MiB/s wr, 1.37k op/s 2024-03-28T18:46:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:56 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:46:59.288 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:46:59 smithi145 ceph-mon[32151]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.36k op/s 2024-03-28T18:46:59.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:46:59 smithi192 ceph-mon[36589]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.36k op/s 2024-03-28T18:46:59.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:46:59 smithi164 ceph-mon[36248]: pgmap v627: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 676 MiB/s wr, 1.36k op/s 2024-03-28T18:47:01.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:01 smithi145 ceph-mon[32151]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 659 MiB/s wr, 1.39k op/s 2024-03-28T18:47:01.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:01 smithi192 ceph-mon[36589]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 659 MiB/s wr, 1.39k op/s 2024-03-28T18:47:01.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:01 smithi164 ceph-mon[36248]: pgmap v628: 129 pgs: 129 active+clean; 10 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 659 MiB/s wr, 1.39k op/s 2024-03-28T18:47:03.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:03 smithi192 ceph-mon[36589]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 607 MiB/s wr, 1.42k op/s 2024-03-28T18:47:03.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:03 smithi164 ceph-mon[36248]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 607 MiB/s wr, 1.42k op/s 2024-03-28T18:47:03.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:03 smithi145 ceph-mon[32151]: pgmap v629: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 607 MiB/s wr, 1.42k op/s 2024-03-28T18:47:05.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:05 smithi192 ceph-mon[36589]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 647 MiB/s wr, 1.61k op/s 2024-03-28T18:47:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:05 smithi164 ceph-mon[36248]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 647 MiB/s wr, 1.61k op/s 2024-03-28T18:47:05.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:05 smithi145 ceph-mon[32151]: pgmap v630: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 647 MiB/s wr, 1.61k op/s 2024-03-28T18:47:07.345 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:07 smithi145 ceph-mon[32151]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 612 MiB/s wr, 1.56k op/s 2024-03-28T18:47:07.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:07 smithi192 ceph-mon[36589]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 612 MiB/s wr, 1.56k op/s 2024-03-28T18:47:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:07 smithi164 ceph-mon[36248]: pgmap v631: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 612 MiB/s wr, 1.56k op/s 2024-03-28T18:47:09.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:09 smithi192 ceph-mon[36589]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 602 MiB/s wr, 1.41k op/s 2024-03-28T18:47:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:09 smithi164 ceph-mon[36248]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 602 MiB/s wr, 1.41k op/s 2024-03-28T18:47:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:09 smithi145 ceph-mon[32151]: pgmap v632: 129 pgs: 129 active+clean; 10 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 602 MiB/s wr, 1.41k op/s 2024-03-28T18:47:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.013+0000 7f9dac412640 1 -- 172.21.15.145:0/937594728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 msgr2=0x7f9da410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.013+0000 7f9dac412640 1 --2- 172.21.15.145:0/937594728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da410de40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c009ca0 tx=0x7f9d9c02c140 comp rx=0 tx=0).stop 2024-03-28T18:47:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.014+0000 7f9dac412640 1 -- 172.21.15.145:0/937594728 shutdown_connections 2024-03-28T18:47:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.014+0000 7f9dac412640 1 --2- 172.21.15.145:0/937594728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da410de40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.014+0000 7f9dac412640 1 --2- 172.21.15.145:0/937594728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da410b010 0x7f9da410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.014+0000 7f9dac412640 1 --2- 172.21.15.145:0/937594728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9da41086a0 0x7f9da410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.014+0000 7f9dac412640 1 -- 172.21.15.145:0/937594728 >> 172.21.15.145:0/937594728 conn(0x7f9da406e6e0 msgr2=0x7f9da4070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.021+0000 7f9dac412640 1 -- 172.21.15.145:0/937594728 shutdown_connections 2024-03-28T18:47:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.027+0000 7f9dac412640 1 -- 172.21.15.145:0/937594728 wait complete. 2024-03-28T18:47:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.028+0000 7f9dac412640 1 Processor -- start 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 -- start start 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9da410b010 0x7f9da4119df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da4117f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4110820 con 0x7f9da410d9c0 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da41106a0 con 0x7f9da410b010 2024-03-28T18:47:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.029+0000 7f9dac412640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da41109a0 con 0x7f9da41086a0 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.030+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.030+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:53630/0 (socket says 172.21.15.145:53630) 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.030+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1830022868 learned_addr learned my addr 172.21.15.145:0/1830022868 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.030+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da4117f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1830022868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9da410b010 msgr2=0x7f9da4119df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9da410b010 0x7f9da4119df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 msgr2=0x7f9da4117f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da4117f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1830022868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da41186b0 con 0x7f9da41086a0 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.031+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b450 tx=0x7f9da000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.032+0000 7f9d9b7fe640 1 -- 172.21.15.145:0/1830022868 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0015020 con 0x7f9da41086a0 2024-03-28T18:47:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.032+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da41189a0 con 0x7f9da41086a0 2024-03-28T18:47:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.032+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da4079980 con 0x7f9da41086a0 2024-03-28T18:47:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.032+0000 7f9d9b7fe640 1 -- 172.21.15.145:0/1830022868 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9da0011d70 con 0x7f9da41086a0 2024-03-28T18:47:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.032+0000 7f9d9b7fe640 1 -- 172.21.15.145:0/1830022868 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da000f340 con 0x7f9da41086a0 2024-03-28T18:47:10.038 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.038+0000 7f9daf001740 1 Processor -- start 2024-03-28T18:47:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.039+0000 7f9d9b7fe640 1 -- 172.21.15.145:0/1830022868 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9da0018020 con 0x7f9da41086a0 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 -- start start 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9d9b7fe640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 0x7f9d900a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5cd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c52ab10 con 0x56142c5c6310 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c4d3900 con 0x56142c5cb480 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daf001740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c526bb0 con 0x56142c5c5920 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:53640/0 (socket says 172.21.15.145:53640) 2024-03-28T18:47:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9daa1ae640 1 -- 172.21.15.145:0/3847897337 learned_addr learned my addr 172.21.15.145:0/3847897337 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.044+0000 7f9d9b7fe640 1 -- 172.21.15.145:0/1830022868 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f9da00ce840 con 0x7f9da41086a0 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 -- 172.21.15.145:0/3847897337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 msgr2=0x56142c5cd910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5cd910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 -- 172.21.15.145:0/3847897337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 msgr2=0x56142c5cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 -- 172.21.15.145:0/3847897337 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56142c23f210 con 0x56142c5c5920 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.045+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9da00981f0 tx=0x7f9da0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8504a2afd4da27 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.046+0000 7f9d98bfe640 1 -- 172.21.15.145:0/3847897337 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da009c490 con 0x56142c5c5920 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.046+0000 7f9d98bfe640 1 -- 172.21.15.145:0/3847897337 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9da0067e90 con 0x56142c5c5920 2024-03-28T18:47:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.046+0000 7f9d98bfe640 1 -- 172.21.15.145:0/3847897337 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0011890 con 0x56142c5c5920 2024-03-28T18:47:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.047+0000 7f9daf001740 1 -- 172.21.15.145:0/3847897337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 msgr2=0x56142c5c5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.047+0000 7f9daf001740 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9da00981f0 tx=0x7f9da0008030 comp rx=0 tx=0).stop 2024-03-28T18:47:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.048+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 0x7f9d900a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.052+0000 7f9daf001740 1 -- 172.21.15.145:0/3847897337 shutdown_connections 2024-03-28T18:47:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.052+0000 7f9daf001740 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.052+0000 7f9daf001740 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.052+0000 7f9daf001740 1 --2- 172.21.15.145:0/3847897337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5c5d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.052+0000 7f9daf001740 1 -- 172.21.15.145:0/3847897337 >> 172.21.15.145:0/3847897337 conn(0x56142c5c0e90 msgr2=0x56142c5c1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.059+0000 7f9daf001740 1 -- 172.21.15.145:0/3847897337 shutdown_connections 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.063+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 0x7f9d900a4fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c032630 tx=0x7f9d9c0323e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.063+0000 7f9daf001740 1 -- 172.21.15.145:0/3847897337 wait complete. 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.063+0000 7f9daf001740 1 -- start start 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.063+0000 7f9daf001740 1 Processor -- start 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5d7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c5ce430 con 0x56142c5c6310 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c4e6d50 con 0x56142c5cb480 2024-03-28T18:47:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56142c5ce5b0 con 0x56142c5c5920 2024-03-28T18:47:10.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.064+0000 7f9daf001740 10 client.0 _ll_register_callbacks cb 0x56142c4e6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:10.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.065+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.065+0000 7f9daa1ae640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:53650/0 (socket says 172.21.15.145:53650) 2024-03-28T18:47:10.065 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.065+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1735291556 learned_addr learned my addr 172.21.15.145:0/1735291556 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1735291556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 msgr2=0x56142c5d7180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5d7180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1735291556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 msgr2=0x56142c5d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 -- 172.21.15.145:0/1735291556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56142c30b620 con 0x56142c5c5920 2024-03-28T18:47:10.066 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9da0000c00 tx=0x7f9da0011c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a50742c1eb4506e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.067 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9d8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:47:10.067 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da00b7020 con 0x56142c5c5920 2024-03-28T18:47:10.067 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.066+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f9da0011890 con 0x56142c5c5920 2024-03-28T18:47:10.068 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.068+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9da0067e90 con 0x56142c5c5920 2024-03-28T18:47:10.069 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.069+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9da00a32d0 con 0x56142c5c5920 2024-03-28T18:47:10.072 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 10 client.25822 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56142c5ce430 con 0x56142c5c5920 2024-03-28T18:47:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9da0067e90 con 0x56142c5c5920 2024-03-28T18:47:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f9da004e020 con 0x56142c5c5920 2024-03-28T18:47:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 10 client.25822 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:10.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 10 client.25822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:10.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 20 client.25822 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:10.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5dbcf0 0x56142c5de110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 4 client.25822 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:10.074 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.073+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56142c538050 con 0x56142c5dbcf0 2024-03-28T18:47:10.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.075+0000 7f9da99ad640 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5dbcf0 0x56142c5de110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.078 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.078+0000 7f9da99ad640 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5dbcf0 0x56142c5de110 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca774eff9421033 server_cookie=b509e6d3e5fdfc2c in_seq=0 out_seq=0 2024-03-28T18:47:10.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.081+0000 7f9d8dffb640 10 client.25822 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:10.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.081+0000 7f9d8dffb640 1 -- 172.21.15.145:0/1735291556 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56142c538050 con 0x56142c5dbcf0 2024-03-28T18:47:10.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.081+0000 7f9d8dffb640 10 client.25822 handle_command_reply: tid=0 2024-03-28T18:47:10.082 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.081+0000 7f9daf001740 1 client.25822 shutdown 2024-03-28T18:47:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.089+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 msgr2=0x56142c5cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.089+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9da0000c00 tx=0x7f9da0011c00 comp rx=0 tx=0).stop 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 shutdown_connections 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5dbcf0 0x56142c5de110 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5cb480 0x56142c5d7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 --2- 172.21.15.145:0/1735291556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c5920 0x56142c5cfc70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.093+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 >> 172.21.15.145:0/1735291556 conn(0x56142c5c0e90 msgr2=0x56142c5c0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:10.093 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.094+0000 7f9d8dffb640 0 client.25822 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:10.096 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.096+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 shutdown_connections 2024-03-28T18:47:10.096 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.096+0000 7f9daf001740 1 -- 172.21.15.145:0/1735291556 wait complete. 2024-03-28T18:47:10.100 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.100+0000 7f9daf001740 20 client.25822 trim_cache size 0 max 16384 2024-03-28T18:47:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.130+0000 7f9daf001740 1 Processor -- start 2024-03-28T18:47:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 -- start start 2024-03-28T18:47:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c5920 0x56142c5d7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8800faf0 con 0x56142c5c5920 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88011470 con 0x56142c5c6310 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.131+0000 7f9daf001740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88010150 con 0x56142c5c0e90 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:37560/0 (socket says 172.21.15.145:37560) 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 -- 172.21.15.145:0/1006762797 learned_addr learned my addr 172.21.15.145:0/1006762797 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 -- 172.21.15.145:0/1006762797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 msgr2=0x56142c5c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 -- 172.21.15.145:0/1006762797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c5920 msgr2=0x56142c5d7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c5920 0x56142c5d7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.132+0000 7f9daa9af640 1 -- 172.21.15.145:0/1006762797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56142c5d0d00 con 0x56142c5c6310 2024-03-28T18:47:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.134+0000 7f9daa9af640 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c032ec0 tx=0x7f9d9c03e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcfd5a5d467cc693 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.135+0000 7f9d8ffff640 1 -- 172.21.15.145:0/1006762797 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c044070 con 0x56142c5c6310 2024-03-28T18:47:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.139+0000 7f9d8ffff640 1 -- 172.21.15.145:0/1006762797 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d9c050070 con 0x56142c5c6310 2024-03-28T18:47:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.139+0000 7f9d8ffff640 1 -- 172.21.15.145:0/1006762797 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c039040 con 0x56142c5c6310 2024-03-28T18:47:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.143+0000 7f9daf001740 1 -- 172.21.15.145:0/1006762797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 msgr2=0x56142c5c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.143+0000 7f9daf001740 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c032ec0 tx=0x7f9d9c03e000 comp rx=0 tx=0).stop 2024-03-28T18:47:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.151+0000 7f9daf001740 1 -- 172.21.15.145:0/1006762797 shutdown_connections 2024-03-28T18:47:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.151+0000 7f9daf001740 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c6310 0x56142c5c4630 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.151+0000 7f9daf001740 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c5920 0x56142c5d7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.151+0000 7f9daf001740 1 --2- 172.21.15.145:0/1006762797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.151+0000 7f9daf001740 1 -- 172.21.15.145:0/1006762797 >> 172.21.15.145:0/1006762797 conn(0x56142c5caf50 msgr2=0x56142c5c7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.158+0000 7f9daf001740 1 -- 172.21.15.145:0/1006762797 shutdown_connections 2024-03-28T18:47:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.159+0000 7f9daf001740 1 -- 172.21.15.145:0/1006762797 wait complete. 2024-03-28T18:47:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.159+0000 7f9daf001740 1 -- start start 2024-03-28T18:47:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.159+0000 7f9daf001740 1 Processor -- start 2024-03-28T18:47:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88010e10 con 0x56142c5c6310 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d880107b0 con 0x56142c5c5920 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8800dcb0 con 0x56142c5c0e90 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daf001740 10 client.0 _ll_register_callbacks cb 0x7f9da009bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9da99ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.160+0000 7f9daa9af640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:37568/0 (socket says 172.21.15.145:37568) 2024-03-28T18:47:10.161 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 -- 172.21.15.145:0/2196195722 learned_addr learned my addr 172.21.15.145:0/2196195722 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 -- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 msgr2=0x56142c5d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5d43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 -- 172.21.15.145:0/2196195722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 msgr2=0x56142c5d5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 -- 172.21.15.145:0/2196195722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56142c5d5f10 con 0x56142c5c5920 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.161+0000 7f9da99ad640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x56142c5c6710 tx=0x7f9d94016f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67a73f5e8c03612 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.162+0000 7f9d8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.162+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94016030 con 0x56142c5c5920 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.162+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f9d940161d0 con 0x56142c5c5920 2024-03-28T18:47:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.162+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d9c050070 con 0x56142c5c5920 2024-03-28T18:47:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.163+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d94015440 con 0x56142c5c5920 2024-03-28T18:47:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.163+0000 7f9daf001740 10 client.25739 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.163+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56142c5d72c0 con 0x56142c5c5920 2024-03-28T18:47:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.163+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d9c050070 con 0x56142c5c5920 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f9d940296b0 con 0x56142c5c5920 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 10 client.25739 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 10 client.25739 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 20 client.25739 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5cb480 0x56142c639bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 4 client.25739 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:47:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56142c5323a0 con 0x56142c5cb480 2024-03-28T18:47:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.165+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5cb480 0x56142c639bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:10.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.167+0000 7f9daa1ae640 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5cb480 0x56142c639bc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4248b9d86055e1f server_cookie=b2d735a452ab73b0 in_seq=0 out_seq=0 2024-03-28T18:47:10.167 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:10.167+0000 7f9d8dffb640 10 client.25739 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:11 smithi192 ceph-mon[36589]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-03-28T18:47:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:11 smithi164 ceph-mon[36248]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-03-28T18:47:11.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:11 smithi145 ceph-mon[32151]: pgmap v633: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 611 MiB/s wr, 1.22k op/s 2024-03-28T18:47:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:13 smithi192 ceph-mon[36589]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 593 MiB/s wr, 1.06k op/s 2024-03-28T18:47:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:13 smithi164 ceph-mon[36248]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 593 MiB/s wr, 1.06k op/s 2024-03-28T18:47:13.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:13 smithi145 ceph-mon[32151]: pgmap v634: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 593 MiB/s wr, 1.06k op/s 2024-03-28T18:47:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:15 smithi192 ceph-mon[36589]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 750 MiB/s wr, 1.42k op/s 2024-03-28T18:47:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:15 smithi164 ceph-mon[36248]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 750 MiB/s wr, 1.42k op/s 2024-03-28T18:47:15.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:15 smithi145 ceph-mon[32151]: pgmap v635: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 750 MiB/s wr, 1.42k op/s 2024-03-28T18:47:16.218 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9d8dffb640 1 -- 172.21.15.145:0/2196195722 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56142c5323a0 con 0x56142c5cb480 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9d8dffb640 10 client.25739 handle_command_reply: tid=0 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 client.25739 shutdown 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 msgr2=0x56142c5d4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x56142c5c6710 tx=0x7f9d94016f00 comp rx=0 tx=0).stop 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 shutdown_connections 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x56142c5cb480 0x56142c639bc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56142c5c6310 0x56142c5d5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56142c5c5920 0x56142c5d4910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 --2- 172.21.15.145:0/2196195722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56142c5c0e90 0x56142c5d43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 >> 172.21.15.145:0/2196195722 conn(0x56142c5caf50 msgr2=0x56142c5c40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.218+0000 7f9d8dffb640 0 client.25739 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.219+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 shutdown_connections 2024-03-28T18:47:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.219+0000 7f9daf001740 1 -- 172.21.15.145:0/2196195722 wait complete. 2024-03-28T18:47:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.219+0000 7f9daf001740 20 client.25739 trim_cache size 0 max 16384 2024-03-28T18:47:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:47:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:47:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:47:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 msgr2=0x7f9d900a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 0x7f9d900a4fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c032630 tx=0x7f9d9c0323e0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 msgr2=0x7f9da41198b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b450 tx=0x7f9da000b910 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 shutdown_connections 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9da410d9c0 0x7f9da4117f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9da410b010 0x7f9da4119df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f9d900a2ae0 0x7f9d900a4fa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 --2- 172.21.15.145:0/1830022868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9da41086a0 0x7f9da41198b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 >> 172.21.15.145:0/1830022868 conn(0x7f9da406e6e0 msgr2=0x7f9da410b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 shutdown_connections 2024-03-28T18:47:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:16.220+0000 7f9dac412640 1 -- 172.21.15.145:0/1830022868 wait complete. 2024-03-28T18:47:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:17 smithi192 ceph-mon[36589]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 653 MiB/s wr, 1.18k op/s 2024-03-28T18:47:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:17 smithi164 ceph-mon[36248]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 653 MiB/s wr, 1.18k op/s 2024-03-28T18:47:17.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:17 smithi145 ceph-mon[32151]: pgmap v636: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 653 MiB/s wr, 1.18k op/s 2024-03-28T18:47:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:19 smithi192 ceph-mon[36589]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 691 MiB/s wr, 1.28k op/s 2024-03-28T18:47:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:19 smithi164 ceph-mon[36248]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 691 MiB/s wr, 1.28k op/s 2024-03-28T18:47:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:19 smithi145 ceph-mon[32151]: pgmap v637: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 691 MiB/s wr, 1.28k op/s 2024-03-28T18:47:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:21 smithi192 ceph-mon[36589]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 1.40k op/s 2024-03-28T18:47:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:21 smithi164 ceph-mon[36248]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 1.40k op/s 2024-03-28T18:47:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:21 smithi145 ceph-mon[32151]: pgmap v638: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 685 MiB/s wr, 1.40k op/s 2024-03-28T18:47:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:23 smithi192 ceph-mon[36589]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 652 MiB/s wr, 1.40k op/s 2024-03-28T18:47:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:23 smithi164 ceph-mon[36248]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 652 MiB/s wr, 1.40k op/s 2024-03-28T18:47:23.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:23 smithi145 ceph-mon[32151]: pgmap v639: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 652 MiB/s wr, 1.40k op/s 2024-03-28T18:47:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:25 smithi192 ceph-mon[36589]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 809 MiB/s wr, 1.72k op/s 2024-03-28T18:47:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:25 smithi164 ceph-mon[36248]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 809 MiB/s wr, 1.72k op/s 2024-03-28T18:47:25.533 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:25 smithi145 ceph-mon[32151]: pgmap v640: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 809 MiB/s wr, 1.72k op/s 2024-03-28T18:47:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:27 smithi192 ceph-mon[36589]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.28k op/s 2024-03-28T18:47:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:27 smithi164 ceph-mon[36248]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.28k op/s 2024-03-28T18:47:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:27 smithi145 ceph-mon[32151]: pgmap v641: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 643 MiB/s wr, 1.28k op/s 2024-03-28T18:47:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:29 smithi164 ceph-mon[36248]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 1.32k op/s 2024-03-28T18:47:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.490+0000 7f7235030640 1 -- 172.21.15.145:0/2769202609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010d9c0 msgr2=0x7f723010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.490+0000 7f7235030640 1 --2- 172.21.15.145:0/2769202609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010d9c0 0x7f723010de40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7228009be0 tx=0x7f722802bf10 comp rx=0 tx=0).stop 2024-03-28T18:47:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.492+0000 7f7235030640 1 -- 172.21.15.145:0/2769202609 shutdown_connections 2024-03-28T18:47:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.492+0000 7f7235030640 1 --2- 172.21.15.145:0/2769202609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010d9c0 0x7f723010de40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.492+0000 7f7235030640 1 --2- 172.21.15.145:0/2769202609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f723010b010 0x7f723010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.492+0000 7f7235030640 1 --2- 172.21.15.145:0/2769202609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72301086a0 0x7f723010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.494 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.492+0000 7f7235030640 1 -- 172.21.15.145:0/2769202609 >> 172.21.15.145:0/2769202609 conn(0x7f723006e6e0 msgr2=0x7f7230070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.496+0000 7f7235030640 1 -- 172.21.15.145:0/2769202609 shutdown_connections 2024-03-28T18:47:29.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.497+0000 7f7235030640 1 -- 172.21.15.145:0/2769202609 wait complete. 2024-03-28T18:47:29.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.498+0000 7f7235030640 1 Processor -- start 2024-03-28T18:47:29.499 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.499+0000 7f7235030640 1 -- start start 2024-03-28T18:47:29.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.499+0000 7f7235030640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72301086a0 0x7f72301a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.499+0000 7f7235030640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010b010 0x7f72301a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f7235030640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f7235030640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723011c180 con 0x7f72301086a0 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f7235030640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723011c000 con 0x7f723010b010 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f722f59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f722f59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:58342/0 (socket says 172.21.15.145:58342) 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f722f59e640 1 -- 172.21.15.145:0/500792551 learned_addr learned my addr 172.21.15.145:0/500792551 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.500+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723011c300 con 0x7f723010d9c0 2024-03-28T18:47:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.504+0000 7f722e59c640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010b010 0x7f72301a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.504+0000 7f722ed9d640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72301086a0 0x7f72301a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.505+0000 7f722f59e640 1 -- 172.21.15.145:0/500792551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010b010 msgr2=0x7f72301a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.505+0000 7f722f59e640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010b010 0x7f72301a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.505+0000 7f722f59e640 1 -- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72301086a0 msgr2=0x7f72301a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.505+0000 7f722f59e640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72301086a0 0x7f72301a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.505+0000 7f722f59e640 1 -- 172.21.15.145:0/500792551 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72301ae310 con 0x7f723010d9c0 2024-03-28T18:47:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.507+0000 7f722f59e640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7228036040 tx=0x7f7228033e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.510+0000 7f7217fff640 1 -- 172.21.15.145:0/500792551 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228043020 con 0x7f723010d9c0 2024-03-28T18:47:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.510+0000 7f7217fff640 1 -- 172.21.15.145:0/500792551 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7228034b20 con 0x7f723010d9c0 2024-03-28T18:47:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.510+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72301ae5a0 con 0x7f723010d9c0 2024-03-28T18:47:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.510+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72301aea30 con 0x7f723010d9c0 2024-03-28T18:47:29.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.511+0000 7f7217fff640 1 -- 172.21.15.145:0/500792551 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722803eb90 con 0x7f723010d9c0 2024-03-28T18:47:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.516+0000 7f7237c16740 1 Processor -- start 2024-03-28T18:47:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 -- start start 2024-03-28T18:47:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dae5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daeae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daed910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927da4ab10 con 0x55927dae5920 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927d9f3900 con 0x55927dae6310 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7237c16740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927da46bb0 con 0x55927daeb480 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f7217fff640 1 -- 172.21.15.145:0/500792551 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7228046020 con 0x7f723010d9c0 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f722e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daed910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f722e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daed910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:58354/0 (socket says 172.21.15.145:58354) 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f722e59c640 1 -- 172.21.15.145:0/4000486206 learned_addr learned my addr 172.21.15.145:0/4000486206 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.518+0000 7f722ed9d640 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daeae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.519+0000 7f7217fff640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 0x7f72180a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.519+0000 7f7217fff640 1 -- 172.21.15.145:0/500792551 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f72280f10a0 con 0x7f723010d9c0 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 -- 172.21.15.145:0/4000486206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 msgr2=0x55927daed910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daed910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 -- 172.21.15.145:0/4000486206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 msgr2=0x55927dae5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dae5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 -- 172.21.15.145:0/4000486206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55927d75f210 con 0x55927dae6310 2024-03-28T18:47:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.520+0000 7f722ed9d640 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daeae10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7224009c30 tx=0x7f7224011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc4f80f4bdfbc63c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.522+0000 7f72157fa640 1 -- 172.21.15.145:0/4000486206 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722401e070 con 0x55927dae6310 2024-03-28T18:47:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.522+0000 7f72157fa640 1 -- 172.21.15.145:0/4000486206 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72240035f0 con 0x55927dae6310 2024-03-28T18:47:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.522+0000 7f72157fa640 1 -- 172.21.15.145:0/4000486206 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224019690 con 0x55927dae6310 2024-03-28T18:47:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.523+0000 7f722f59e640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 0x7f72180a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.523+0000 7f7237c16740 1 -- 172.21.15.145:0/4000486206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 msgr2=0x55927daeae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.523+0000 7f7237c16740 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daeae10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7224009c30 tx=0x7f7224011810 comp rx=0 tx=0).stop 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.523+0000 7f722f59e640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 0x7f72180a4fa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7228034eb0 tx=0x7f7228033f80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.523+0000 7f7237c16740 1 -- 172.21.15.145:0/4000486206 shutdown_connections 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.524+0000 7f7237c16740 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daed910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.524+0000 7f7237c16740 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daeae10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.524+0000 7f7237c16740 1 --2- 172.21.15.145:0/4000486206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dae5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.524+0000 7f7237c16740 1 -- 172.21.15.145:0/4000486206 >> 172.21.15.145:0/4000486206 conn(0x55927dae0e90 msgr2=0x55927dae1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.525+0000 7f7237c16740 1 -- 172.21.15.145:0/4000486206 shutdown_connections 2024-03-28T18:47:29.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:29 smithi145 ceph-mon[32151]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 1.32k op/s 2024-03-28T18:47:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.527+0000 7f7237c16740 1 -- 172.21.15.145:0/4000486206 wait complete. 2024-03-28T18:47:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.527+0000 7f7237c16740 1 -- start start 2024-03-28T18:47:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.527+0000 7f7237c16740 1 Processor -- start 2024-03-28T18:47:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daefcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 0x55927daf0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927daee430 con 0x55927daeb480 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927da06d50 con 0x55927dae5920 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55927daee5b0 con 0x55927dae6310 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f7237c16740 10 client.0 _ll_register_callbacks cb 0x55927da06720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f722ed9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 0x55927daf0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f722e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daefcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f722ed9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 0x55927daf0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:58370/0 (socket says 172.21.15.145:58370) 2024-03-28T18:47:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.528+0000 7f722ed9d640 1 -- 172.21.15.145:0/2812682873 learned_addr learned my addr 172.21.15.145:0/2812682873 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 -- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 msgr2=0x55927daf0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 0x55927daf0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 -- 172.21.15.145:0/2812682873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 msgr2=0x55927daf7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 -- 172.21.15.145:0/2812682873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55927d82b620 con 0x55927dae5920 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.529+0000 7f722e59c640 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daefcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7220006fa0 tx=0x7f72200346b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2179cc3700ed36a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.530+0000 7f7212ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.530+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722003b020 con 0x55927dae5920 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.531+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72240035f0 con 0x55927dae5920 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.531+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f7220037460 con 0x55927dae5920 2024-03-28T18:47:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.531+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7220046050 con 0x55927dae5920 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.531+0000 7f7237c16740 10 client.25751 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.531+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55927dafa660 con 0x55927dae5920 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72240035f0 con 0x55927dae5920 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f7220036840 con 0x55927dae5920 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 10 client.25751 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 20 client.25751 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db0d250 0x55927db0f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 4 client.25751 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:29.537 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.532+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55927da58050 con 0x55927db0d250 2024-03-28T18:47:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.536+0000 7f722ed9d640 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db0d250 0x55927db0f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.536+0000 7f722ed9d640 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db0d250 0x55927db0f680 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6c743c3cd49ab55 server_cookie=430f3b40adba6d87 in_seq=0 out_seq=0 2024-03-28T18:47:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.536+0000 7f7212ff5640 10 client.25751 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.546+0000 7f7212ff5640 1 -- 172.21.15.145:0/2812682873 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55927da58050 con 0x55927db0d250 2024-03-28T18:47:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.546+0000 7f7212ff5640 10 client.25751 handle_command_reply: tid=0 2024-03-28T18:47:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.547+0000 7f7237c16740 1 client.25751 shutdown 2024-03-28T18:47:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.547+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 msgr2=0x55927daefcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.547+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daefcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7220006fa0 tx=0x7f72200346b0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 shutdown_connections 2024-03-28T18:47:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db0d250 0x55927db0f680 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927dae6310 0x55927daf0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 --2- 172.21.15.145:0/2812682873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daefcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 >> 172.21.15.145:0/2812682873 conn(0x55927dae0e90 msgr2=0x55927dae0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.548+0000 7f7212ff5640 0 client.25751 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.552+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 shutdown_connections 2024-03-28T18:47:29.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.555+0000 7f7237c16740 1 -- 172.21.15.145:0/2812682873 wait complete. 2024-03-28T18:47:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.556+0000 7f7237c16740 20 client.25751 trim_cache size 0 max 16384 2024-03-28T18:47:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.606+0000 7f7237c16740 1 Processor -- start 2024-03-28T18:47:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.606+0000 7f7237c16740 1 -- start start 2024-03-28T18:47:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.607+0000 7f7237c16740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dadbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.607+0000 7f722e59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dadbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.607+0000 7f722e59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dadbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60686/0 (socket says 172.21.15.145:60686) 2024-03-28T18:47:29.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.607+0000 7f722e59c640 1 -- 172.21.15.145:0/373572351 learned_addr learned my addr 172.21.15.145:0/373572351 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:29.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.607+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daf7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.608+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.609 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.608+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c00fb40 con 0x55927dae5920 2024-03-28T18:47:29.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.608+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0114c0 con 0x55927dae6310 2024-03-28T18:47:29.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.608+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0101a0 con 0x55927daeb480 2024-03-28T18:47:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.611+0000 7f722e59c640 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.614+0000 7f722ed9d640 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daf7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.615+0000 7f722ed9d640 1 -- 172.21.15.145:0/373572351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 msgr2=0x55927daef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.615+0000 7f722ed9d640 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daef8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.615+0000 7f722ed9d640 1 -- 172.21.15.145:0/373572351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 msgr2=0x55927dadbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.615+0000 7f722ed9d640 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dadbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.615+0000 7f722ed9d640 1 -- 172.21.15.145:0/373572351 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55927dae66f0 con 0x55927dae6310 2024-03-28T18:47:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.616+0000 7f722ed9d640 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daf7210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7224006010 tx=0x7f7224009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=291438be38e24e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.624+0000 7f7214ff9640 1 -- 172.21.15.145:0/373572351 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722401e040 con 0x55927dae6310 2024-03-28T18:47:29.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.624+0000 7f7214ff9640 1 -- 172.21.15.145:0/373572351 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f722402a1d0 con 0x55927dae6310 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.625+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 msgr2=0x55927daf7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daf7210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7224006010 tx=0x7f7224009ed0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 shutdown_connections 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daeb480 0x55927daef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae6310 0x55927daf7210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 --2- 172.21.15.145:0/373572351 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927dae5920 0x55927dadbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.626+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 >> 172.21.15.145:0/373572351 conn(0x55927dae0e90 msgr2=0x55927dae1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.627+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 shutdown_connections 2024-03-28T18:47:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.627+0000 7f7237c16740 1 -- 172.21.15.145:0/373572351 wait complete. 2024-03-28T18:47:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.627+0000 7f7237c16740 1 -- start start 2024-03-28T18:47:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.627+0000 7f7237c16740 1 Processor -- start 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daf4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daf5ce0 0x55927daf6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c010e60 con 0x55927daeb480 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c010800 con 0x55927dae5920 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c00dd00 con 0x55927daf5ce0 2024-03-28T18:47:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.628+0000 7f7237c16740 10 client.0 _ll_register_callbacks cb 0x7f722000aab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.633+0000 7f722e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daf4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.633+0000 7f722e59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daf5ce0 0x55927daf6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.634+0000 7f722e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daf4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:40884/0 (socket says 172.21.15.145:40884) 2024-03-28T18:47:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.634+0000 7f722e59c640 1 -- 172.21.15.145:0/3999654869 learned_addr learned my addr 172.21.15.145:0/3999654869 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:29.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.643+0000 7f722ed9d640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.652+0000 7f722ed9d640 1 -- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daf5ce0 msgr2=0x55927daf6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.652+0000 7f722ed9d640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daf5ce0 0x55927daf6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.652+0000 7f722ed9d640 1 -- 172.21.15.145:0/3999654869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 msgr2=0x55927daf4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.652+0000 7f722ed9d640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daf4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:29.654 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.652+0000 7f722ed9d640 1 -- 172.21.15.145:0/3999654869 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55927daf66a0 con 0x55927daeb480 2024-03-28T18:47:29.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:29 smithi192 ceph-mon[36589]: pgmap v642: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 668 MiB/s wr, 1.32k op/s 2024-03-28T18:47:29.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.662+0000 7f722ed9d640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf4cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x55927daf1160 tx=0x7f7224005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b97d7972d63937 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.663+0000 7f7212ff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:47:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.667+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722401e040 con 0x55927daeb480 2024-03-28T18:47:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.667+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f72240035f0 con 0x55927daeb480 2024-03-28T18:47:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.667+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55927da523a0 con 0x55927daeb480 2024-03-28T18:47:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.670+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7224003d20 con 0x55927daeb480 2024-03-28T18:47:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.670+0000 7f7237c16740 10 client.15957 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.670+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55927db68e50 con 0x55927daeb480 2024-03-28T18:47:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.672+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55927da523a0 con 0x55927daeb480 2024-03-28T18:47:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.673+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f722402ac50 con 0x55927daeb480 2024-03-28T18:47:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.673+0000 7f7237c16740 10 client.15957 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.674+0000 7f7237c16740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:29.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.674+0000 7f7237c16740 20 client.15957 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.674+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db69660 0x55927db6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.674+0000 7f7237c16740 4 client.15957 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:47:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.674+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55927d9d0710 con 0x55927db69660 2024-03-28T18:47:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.679+0000 7f722e59c640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db69660 0x55927db6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.691+0000 7f722e59c640 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db69660 0x55927db6ba80 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca297f76ac3bdb9c server_cookie=52ef1022ce5deec9 in_seq=0 out_seq=0 2024-03-28T18:47:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:29.691+0000 7f7212ff5640 10 client.15957 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:31.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:31 smithi164 ceph-mon[36248]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 722 MiB/s wr, 1.45k op/s 2024-03-28T18:47:31.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:31 smithi145 ceph-mon[32151]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 722 MiB/s wr, 1.45k op/s 2024-03-28T18:47:31.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:31 smithi192 ceph-mon[36589]: pgmap v643: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 722 MiB/s wr, 1.45k op/s 2024-03-28T18:47:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7212ff5640 1 -- 172.21.15.145:0/3999654869 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55927d9d0710 con 0x55927db69660 2024-03-28T18:47:32.335 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7212ff5640 10 client.15957 handle_command_reply: tid=0 2024-03-28T18:47:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 client.15957 shutdown 2024-03-28T18:47:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 msgr2=0x55927daf4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf4cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x55927daf1160 tx=0x7f7224005e80 comp rx=0 tx=0).stop 2024-03-28T18:47:32.336 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 shutdown_connections 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55927db69660 0x55927db6ba80 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55927daf5ce0 0x55927daf6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55927daeb480 0x55927daf4cd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 --2- 172.21.15.145:0/3999654869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55927dae5920 0x55927daf4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.335+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 >> 172.21.15.145:0/3999654869 conn(0x55927dae0e90 msgr2=0x55927dae1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.336+0000 7f7212ff5640 0 client.15957 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.336+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 shutdown_connections 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.336+0000 7f7237c16740 1 -- 172.21.15.145:0/3999654869 wait complete. 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.336+0000 7f7237c16740 20 client.15957 trim_cache size 0 max 16384 2024-03-28T18:47:32.337 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:47:32.338 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:47:32.338 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:47:32.338 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:47:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.338+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 msgr2=0x7f72180a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.338+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 0x7f72180a4fa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7228034eb0 tx=0x7f7228033f80 comp rx=0 tx=0).stop 2024-03-28T18:47:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.338+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 msgr2=0x7f72301adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.338+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7228036040 tx=0x7f7228033e10 comp rx=0 tx=0).stop 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 shutdown_connections 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f72180a2ae0 0x7f72180a4fa0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f723010d9c0 0x7f72301adca0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723010b010 0x7f72301a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 --2- 172.21.15.145:0/500792551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72301086a0 0x7f72301a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 >> 172.21.15.145:0/500792551 conn(0x7f723006e6e0 msgr2=0x7f7230109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 shutdown_connections 2024-03-28T18:47:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:32.339+0000 7f7235030640 1 -- 172.21.15.145:0/500792551 wait complete. 2024-03-28T18:47:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:33 smithi164 ceph-mon[36248]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 655 MiB/s wr, 1.33k op/s 2024-03-28T18:47:33.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:33 smithi145 ceph-mon[32151]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 655 MiB/s wr, 1.33k op/s 2024-03-28T18:47:33.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:33 smithi192 ceph-mon[36589]: pgmap v644: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 655 MiB/s wr, 1.33k op/s 2024-03-28T18:47:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:35 smithi164 ceph-mon[36248]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 796 MiB/s wr, 1.59k op/s 2024-03-28T18:47:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:35 smithi145 ceph-mon[32151]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 796 MiB/s wr, 1.59k op/s 2024-03-28T18:47:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:35 smithi192 ceph-mon[36589]: pgmap v645: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 796 MiB/s wr, 1.59k op/s 2024-03-28T18:47:37.458 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:37 smithi192 ceph-mon[36589]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 613 MiB/s wr, 1.23k op/s 2024-03-28T18:47:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:37 smithi164 ceph-mon[36248]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 613 MiB/s wr, 1.23k op/s 2024-03-28T18:47:37.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:37 smithi145 ceph-mon[32151]: pgmap v646: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 613 MiB/s wr, 1.23k op/s 2024-03-28T18:47:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:39 smithi164 ceph-mon[36248]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 631 MiB/s wr, 1.20k op/s 2024-03-28T18:47:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:39 smithi145 ceph-mon[32151]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 631 MiB/s wr, 1.20k op/s 2024-03-28T18:47:39.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:39 smithi192 ceph-mon[36589]: pgmap v647: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 631 MiB/s wr, 1.20k op/s 2024-03-28T18:47:41.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:41 smithi145 ceph-mon[32151]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 739 MiB/s wr, 1.30k op/s 2024-03-28T18:47:41.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:41 smithi192 ceph-mon[36589]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 739 MiB/s wr, 1.30k op/s 2024-03-28T18:47:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:41 smithi164 ceph-mon[36248]: pgmap v648: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 739 MiB/s wr, 1.30k op/s 2024-03-28T18:47:43.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:43 smithi164 ceph-mon[36248]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 652 MiB/s wr, 1.08k op/s 2024-03-28T18:47:43.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:43 smithi145 ceph-mon[32151]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 652 MiB/s wr, 1.08k op/s 2024-03-28T18:47:43.658 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:43 smithi192 ceph-mon[36589]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 652 MiB/s wr, 1.08k op/s 2024-03-28T18:47:45.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:45 smithi145 ceph-mon[32151]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 763 MiB/s wr, 1.50k op/s 2024-03-28T18:47:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.585+0000 7f16f49b8640 1 -- 172.21.15.145:0/4075791492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0072be0 msgr2=0x7f16f0073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.585+0000 7f16f49b8640 1 --2- 172.21.15.145:0/4075791492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0072be0 0x7f16f0073060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009900 tx=0x7f16e402bc80 comp rx=0 tx=0).stop 2024-03-28T18:47:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.586+0000 7f16f49b8640 1 -- 172.21.15.145:0/4075791492 shutdown_connections 2024-03-28T18:47:45.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.586+0000 7f16f49b8640 1 --2- 172.21.15.145:0/4075791492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f010f9e0 0x7f16f0111dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.586+0000 7f16f49b8640 1 --2- 172.21.15.145:0/4075791492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0072be0 0x7f16f0073060 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009900 tx=0x7f16e402bc80 comp rx=0 tx=0).stop 2024-03-28T18:47:45.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.586+0000 7f16f49b8640 1 --2- 172.21.15.145:0/4075791492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f0109280 0x7f16f00726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.586+0000 7f16f49b8640 1 -- 172.21.15.145:0/4075791492 >> 172.21.15.145:0/4075791492 conn(0x7f16f006e510 msgr2=0x7f16f0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.593+0000 7f16f49b8640 1 -- 172.21.15.145:0/4075791492 shutdown_connections 2024-03-28T18:47:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.594+0000 7f16f49b8640 1 -- 172.21.15.145:0/4075791492 wait complete. 2024-03-28T18:47:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.595+0000 7f16f49b8640 1 Processor -- start 2024-03-28T18:47:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.596+0000 7f16f49b8640 1 -- start start 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.596+0000 7f16f49b8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0109280 0x7f16f01a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16f49b8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 0x7f16f01a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16f49b8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16f49b8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f01159a0 con 0x7f16f010f9e0 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16f49b8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0115820 con 0x7f16f01a57c0 2024-03-28T18:47:45.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16f49b8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0115b20 con 0x7f16f0109280 2024-03-28T18:47:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16ee59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 0x7f16f01a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.597+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.598+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52240/0 (socket says 172.21.15.145:52240) 2024-03-28T18:47:45.598 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.598+0000 7f16ef59e640 1 -- 172.21.15.145:0/3055332444 learned_addr learned my addr 172.21.15.145:0/3055332444 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.598+0000 7f16ef59e640 1 -- 172.21.15.145:0/3055332444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0109280 msgr2=0x7f16f01a0220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ef59e640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0109280 0x7f16f01a0220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ef59e640 1 -- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 msgr2=0x7f16f01a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ef59e640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 0x7f16f01a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ef59e640 1 -- 172.21.15.145:0/3055332444 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f01a8220 con 0x7f16f01a57c0 2024-03-28T18:47:45.599 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ee59c640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 0x7f16f01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:45.600 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.599+0000 7f16ef59e640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f16e800eb10 tx=0x7f16e800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.607+0000 7f16d7fff640 1 -- 172.21.15.145:0/3055332444 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8004b20 con 0x7f16f01a57c0 2024-03-28T18:47:45.607 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.607+0000 7f16d7fff640 1 -- 172.21.15.145:0/3055332444 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e8004cc0 con 0x7f16f01a57c0 2024-03-28T18:47:45.608 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.607+0000 7f16d7fff640 1 -- 172.21.15.145:0/3055332444 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8010790 con 0x7f16f01a57c0 2024-03-28T18:47:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.610+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f01a8510 con 0x7f16f01a57c0 2024-03-28T18:47:45.613 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.610+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f01a89e0 con 0x7f16f01a57c0 2024-03-28T18:47:45.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.614+0000 7f16f7da3740 1 Processor -- start 2024-03-28T18:47:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 -- start start 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b1f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b22380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1a76b10 con 0x55fda1b1ff10 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1a1f900 con 0x55fda1b1a920 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16f7da3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1a72bb0 con 0x55fda1b1b310 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.618+0000 7f16ee59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.619+0000 7f16ee59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52250/0 (socket says 172.21.15.145:52250) 2024-03-28T18:47:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.619+0000 7f16ee59c640 1 -- 172.21.15.145:0/7445362 learned_addr learned my addr 172.21.15.145:0/7445362 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:45.620 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.619+0000 7f16eed9d640 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b1f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.621+0000 7f16d7fff640 1 -- 172.21.15.145:0/3055332444 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f16e8005020 con 0x7f16f01a57c0 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.622+0000 7f16ee59c640 1 -- 172.21.15.145:0/7445362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 msgr2=0x55fda1b1f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.622+0000 7f16ee59c640 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b1f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.622+0000 7f16ee59c640 1 -- 172.21.15.145:0/7445362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 msgr2=0x55fda1b22380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.622+0000 7f16ee59c640 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b22380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.622+0000 7f16ee59c640 1 -- 172.21.15.145:0/7445362 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fda178b210 con 0x55fda1b1a920 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.623+0000 7f16d7fff640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 0x7f16d80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.623+0000 7f16d7fff640 1 -- 172.21.15.145:0/3055332444 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f16e80cf280 con 0x7f16f01a57c0 2024-03-28T18:47:45.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.625+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 0x7f16d80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.627+0000 7f16ee59c640 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16e402c290 tx=0x7f16e4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77032c4b06ce96c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.631+0000 7f16d57fa640 1 -- 172.21.15.145:0/7445362 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4044020 con 0x55fda1b1a920 2024-03-28T18:47:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.631+0000 7f16d57fa640 1 -- 172.21.15.145:0/7445362 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e402cb00 con 0x55fda1b1a920 2024-03-28T18:47:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.632+0000 7f16f7da3740 1 -- 172.21.15.145:0/7445362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 msgr2=0x55fda1b1ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.632+0000 7f16f7da3740 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16e402c290 tx=0x7f16e4009560 comp rx=0 tx=0).stop 2024-03-28T18:47:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.633+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 0x7f16d80a4ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55fda1b1b7b0 tx=0x7f16e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.634+0000 7f16f7da3740 1 -- 172.21.15.145:0/7445362 shutdown_connections 2024-03-28T18:47:45.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.634+0000 7f16f7da3740 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b22380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.634+0000 7f16f7da3740 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b1f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.634+0000 7f16f7da3740 1 --2- 172.21.15.145:0/7445362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b1ad40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.634+0000 7f16f7da3740 1 -- 172.21.15.145:0/7445362 >> 172.21.15.145:0/7445362 conn(0x55fda1b0d010 msgr2=0x55fda1b0d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:45.638 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.637+0000 7f16f7da3740 1 -- 172.21.15.145:0/7445362 shutdown_connections 2024-03-28T18:47:45.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.638+0000 7f16f7da3740 1 -- 172.21.15.145:0/7445362 wait complete. 2024-03-28T18:47:45.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.639+0000 7f16f7da3740 1 -- start start 2024-03-28T18:47:45.639 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.639+0000 7f16f7da3740 1 Processor -- start 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b24630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b24b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1b22e90 con 0x55fda1b1ff10 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1a32d50 con 0x55fda1b1a920 2024-03-28T18:47:45.640 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fda1b23010 con 0x55fda1b1b310 2024-03-28T18:47:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16f7da3740 10 client.0 _ll_register_callbacks cb 0x55fda1a32720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35170/0 (socket says 172.21.15.145:35170) 2024-03-28T18:47:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.640+0000 7f16ef59e640 1 -- 172.21.15.145:0/3000600165 learned_addr learned my addr 172.21.15.145:0/3000600165 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b24630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16ee59c640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b24b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 -- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 msgr2=0x55fda1b24b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b24b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 -- 172.21.15.145:0/3000600165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 msgr2=0x55fda1b2bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 -- 172.21.15.145:0/3000600165 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fda1857620 con 0x55fda1b1a920 2024-03-28T18:47:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.641+0000 7f16eed9d640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b24630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55fda1b0df80 tx=0x7f16e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a857d7bf9cf23e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.646+0000 7f16d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.646+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0004220 con 0x55fda1b1a920 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.646+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16e402cb00 con 0x55fda1b1a920 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.646+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f16e00043c0 con 0x55fda1b1a920 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.647+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e002f050 con 0x55fda1b1a920 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.647+0000 7f16f7da3740 10 client.25769 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.647+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fda1b2efc0 con 0x55fda1b1a920 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e402cb00 con 0x55fda1b1a920 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f16e0003c00 con 0x55fda1b1a920 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 10 client.25769 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 10 client.25769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 20 client.25769 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b30bd0 0x55fda1b33030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 4 client.25769 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:45.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.653+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fda1a84050 con 0x55fda1b30bd0 2024-03-28T18:47:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:45 smithi192 ceph-mon[36589]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 763 MiB/s wr, 1.50k op/s 2024-03-28T18:47:45.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.661+0000 7f16ee59c640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b30bd0 0x55fda1b33030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.665+0000 7f16ee59c640 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b30bd0 0x55fda1b33030 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2cfc663c297eda3 server_cookie=549ad93d31eea1a9 in_seq=0 out_seq=0 2024-03-28T18:47:45.665 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.665+0000 7f16d2ff5640 10 client.25769 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:45.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.668+0000 7f16d2ff5640 1 -- 172.21.15.145:0/3000600165 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fda1a84050 con 0x55fda1b30bd0 2024-03-28T18:47:45.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.668+0000 7f16d2ff5640 10 client.25769 handle_command_reply: tid=0 2024-03-28T18:47:45.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.668+0000 7f16f7da3740 1 client.25769 shutdown 2024-03-28T18:47:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.674+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 msgr2=0x55fda1b24630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:45.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.674+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b24630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55fda1b0df80 tx=0x7f16e0004730 comp rx=0 tx=0).stop 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 shutdown_connections 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b30bd0 0x55fda1b33030 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b24b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 --2- 172.21.15.145:0/3000600165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1a920 0x55fda1b24630 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 >> 172.21.15.145:0/3000600165 conn(0x55fda1b0d010 msgr2=0x55fda1b19fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.680+0000 7f16d2ff5640 0 client.25769 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:45.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.688+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 shutdown_connections 2024-03-28T18:47:45.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.690+0000 7f16f7da3740 1 -- 172.21.15.145:0/3000600165 wait complete. 2024-03-28T18:47:45.690 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.690+0000 7f16f7da3740 20 client.25769 trim_cache size 0 max 16384 2024-03-28T18:47:45.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:45 smithi164 ceph-mon[36248]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 763 MiB/s wr, 1.50k op/s 2024-03-28T18:47:45.748 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.748+0000 7f16f7da3740 1 Processor -- start 2024-03-28T18:47:45.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.755+0000 7f16f7da3740 1 -- start start 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.755+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1a920 0x55fda1b1f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.755+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.755+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.756+0000 7f16f7da3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00fb80 con 0x55fda1b1b310 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.756+0000 7f16f7da3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc011500 con 0x55fda1b1ff10 2024-03-28T18:47:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.756+0000 7f16f7da3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0101e0 con 0x55fda1b1a920 2024-03-28T18:47:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.760+0000 7f16ee59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.760+0000 7f16ee59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:35178/0 (socket says 172.21.15.145:35178) 2024-03-28T18:47:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.760+0000 7f16ee59c640 1 -- 172.21.15.145:0/400460391 learned_addr learned my addr 172.21.15.145:0/400460391 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.761+0000 7f16ee59c640 1 -- 172.21.15.145:0/400460391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1a920 msgr2=0x55fda1b1f870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.761+0000 7f16ee59c640 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1a920 0x55fda1b1f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.761+0000 7f16ee59c640 1 -- 172.21.15.145:0/400460391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 msgr2=0x55fda1b2bb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.761+0000 7f16ee59c640 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.761+0000 7f16ee59c640 1 -- 172.21.15.145:0/400460391 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fda1b08410 con 0x55fda1b1b310 2024-03-28T18:47:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.763+0000 7f16ee59c640 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f16e40096c0 tx=0x7f16e402bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e7297ab70e7095 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.765+0000 7f16d4ff9640 1 -- 172.21.15.145:0/400460391 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4056070 con 0x55fda1b1b310 2024-03-28T18:47:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.765+0000 7f16d4ff9640 1 -- 172.21.15.145:0/400460391 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e4051260 con 0x55fda1b1b310 2024-03-28T18:47:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.766+0000 7f16f7da3740 1 -- 172.21.15.145:0/400460391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 msgr2=0x55fda1b23c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.766+0000 7f16f7da3740 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f16e40096c0 tx=0x7f16e402bc80 comp rx=0 tx=0).stop 2024-03-28T18:47:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.766+0000 7f16d4ff9640 1 -- 172.21.15.145:0/400460391 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4044040 con 0x55fda1b1b310 2024-03-28T18:47:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.768+0000 7f16f7da3740 1 -- 172.21.15.145:0/400460391 shutdown_connections 2024-03-28T18:47:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.768+0000 7f16f7da3740 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b2bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.768+0000 7f16f7da3740 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1b310 0x55fda1b23c80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.768+0000 7f16f7da3740 1 --2- 172.21.15.145:0/400460391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1a920 0x55fda1b1f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.768+0000 7f16f7da3740 1 -- 172.21.15.145:0/400460391 >> 172.21.15.145:0/400460391 conn(0x55fda1b0d010 msgr2=0x55fda1b24240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.780+0000 7f16f7da3740 1 -- 172.21.15.145:0/400460391 shutdown_connections 2024-03-28T18:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.781+0000 7f16f7da3740 1 -- 172.21.15.145:0/400460391 wait complete. 2024-03-28T18:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.781+0000 7f16f7da3740 1 -- start start 2024-03-28T18:47:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.781+0000 7f16f7da3740 1 Processor -- start 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1a920 0x55fda1b28a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b28fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc010ea0 con 0x55fda1b1a920 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc010840 con 0x55fda1b1ff10 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00dd40 con 0x55fda1b1b310 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.782+0000 7f16f7da3740 10 client.0 _ll_register_callbacks cb 0x7f16e0012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52274/0 (socket says 172.21.15.145:52274) 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 -- 172.21.15.145:0/2077596132 learned_addr learned my addr 172.21.15.145:0/2077596132 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 -- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 msgr2=0x55fda1b28fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b28fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 -- 172.21.15.145:0/2077596132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1a920 msgr2=0x55fda1b28a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1a920 0x55fda1b28a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.783+0000 7f16ef59e640 1 -- 172.21.15.145:0/2077596132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fda1b8aa30 con 0x55fda1b1ff10 2024-03-28T18:47:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.785+0000 7f16ef59e640 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f16e80049d0 tx=0x7f16e809b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e73002326519c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.790+0000 7f16d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:47:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.790+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e80b3070 con 0x55fda1b1ff10 2024-03-28T18:47:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.790+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f16e80a3d20 con 0x55fda1b1ff10 2024-03-28T18:47:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.791+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16e4051260 con 0x55fda1b1ff10 2024-03-28T18:47:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.792+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e809b810 con 0x55fda1b1ff10 2024-03-28T18:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.793+0000 7f16f7da3740 10 client.25772 fetch_fsmap learned FSMap version 31 2024-03-28T18:47:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.793+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fda1b8bd10 con 0x55fda1b1ff10 2024-03-28T18:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.794+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16e4051260 con 0x55fda1b1ff10 2024-03-28T18:47:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.795+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f16e80a7030 con 0x55fda1b1ff10 2024-03-28T18:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 10 client.25772 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 10 client.25772 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 20 client.25772 populate_metadata read hostname 'smithi145' 2024-03-28T18:47:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b8c650 0x55fda1b8ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 4 client.25772 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:47:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.796+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fda1a7e3a0 con 0x55fda1b8c650 2024-03-28T18:47:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.799+0000 7f16eed9d640 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b8c650 0x55fda1b8ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.802+0000 7f16eed9d640 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b8c650 0x55fda1b8ea70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e65f64af681ca7ac server_cookie=5d8adb6298687a2b in_seq=0 out_seq=0 2024-03-28T18:47:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:45.805+0000 7f16d2ff5640 10 client.25772 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:46.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:46 smithi192 ceph-mon[36589]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 616 MiB/s wr, 1.31k op/s 2024-03-28T18:47:46.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:46 smithi164 ceph-mon[36248]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 616 MiB/s wr, 1.31k op/s 2024-03-28T18:47:46.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:46 smithi145 ceph-mon[32151]: pgmap v651: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 616 MiB/s wr, 1.31k op/s 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16d2ff5640 1 -- 172.21.15.145:0/2077596132 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fda1a7e3a0 con 0x55fda1b8c650 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16d2ff5640 10 client.25772 handle_command_reply: tid=0 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 client.25772 shutdown 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 msgr2=0x55fda1b8a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f16e80049d0 tx=0x7f16e809b350 comp rx=0 tx=0).stop 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 shutdown_connections 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55fda1b8c650 0x55fda1b8ea70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fda1b1ff10 0x55fda1b8a4f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fda1b1b310 0x55fda1b28fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 --2- 172.21.15.145:0/2077596132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55fda1b1a920 0x55fda1b28a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 >> 172.21.15.145:0/2077596132 conn(0x55fda1b0d010 msgr2=0x55fda1b0ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16d2ff5640 0 client.25772 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 shutdown_connections 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.936+0000 7f16f7da3740 1 -- 172.21.15.145:0/2077596132 wait complete. 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.937+0000 7f16f7da3740 20 client.25772 trim_cache size 0 max 16384 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:47:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:47:47.939 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 msgr2=0x7f16d80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 0x7f16d80a4ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55fda1b1b7b0 tx=0x7f16e0005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 msgr2=0x7f16f01a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f16e800eb10 tx=0x7f16e800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 shutdown_connections 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f01a57c0 0x7f16f01a7bb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16f010f9e0 0x7f16f01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f16d80a2a00 0x7f16d80a4ec0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.938+0000 7f16f49b8640 1 --2- 172.21.15.145:0/3055332444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f0109280 0x7f16f01a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:47.941 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.940+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 >> 172.21.15.145:0/3055332444 conn(0x7f16f006e510 msgr2=0x7f16f010b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:47.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.942+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 shutdown_connections 2024-03-28T18:47:47.942 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:47:47.942+0000 7f16f49b8640 1 -- 172.21.15.145:0/3055332444 wait complete. 2024-03-28T18:47:49.276 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:49 smithi145 ceph-mon[32151]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 635 MiB/s wr, 1.37k op/s 2024-03-28T18:47:49.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:49 smithi192 ceph-mon[36589]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 635 MiB/s wr, 1.37k op/s 2024-03-28T18:47:49.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:49 smithi164 ceph-mon[36248]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 635 MiB/s wr, 1.37k op/s 2024-03-28T18:47:51.277 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:51 smithi145 ceph-mon[32151]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 737 MiB/s wr, 1.45k op/s 2024-03-28T18:47:51.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:51 smithi192 ceph-mon[36589]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 737 MiB/s wr, 1.45k op/s 2024-03-28T18:47:51.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:51 smithi164 ceph-mon[36248]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 737 MiB/s wr, 1.45k op/s 2024-03-28T18:47:53.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:53 smithi192 ceph-mon[36589]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 621 MiB/s wr, 1.23k op/s 2024-03-28T18:47:53.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:53 smithi164 ceph-mon[36248]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 621 MiB/s wr, 1.23k op/s 2024-03-28T18:47:53.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:53 smithi145 ceph-mon[32151]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 621 MiB/s wr, 1.23k op/s 2024-03-28T18:47:55.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:55 smithi192 ceph-mon[36589]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 733 MiB/s wr, 1.45k op/s 2024-03-28T18:47:55.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:55 smithi164 ceph-mon[36248]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 733 MiB/s wr, 1.45k op/s 2024-03-28T18:47:55.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:55 smithi145 ceph-mon[32151]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 733 MiB/s wr, 1.45k op/s 2024-03-28T18:47:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:57 smithi192 ceph-mon[36589]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 MiB/s wr, 1.06k op/s 2024-03-28T18:47:57.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:57 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:57 smithi164 ceph-mon[36248]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 MiB/s wr, 1.06k op/s 2024-03-28T18:47:57.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:57 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:57.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:57 smithi145 ceph-mon[32151]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 620 MiB/s wr, 1.06k op/s 2024-03-28T18:47:57.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:57 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:59.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:47:59 smithi192 ceph-mon[36589]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 649 MiB/s wr, 1.10k op/s 2024-03-28T18:47:59.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:47:59 smithi164 ceph-mon[36248]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 649 MiB/s wr, 1.10k op/s 2024-03-28T18:47:59.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:47:59 smithi145 ceph-mon[32151]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 649 MiB/s wr, 1.10k op/s 2024-03-28T18:48:00.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:00.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:00.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:48:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:48:00.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:00.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:00.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:48:01.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:01 smithi192 ceph-mon[36589]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 739 MiB/s wr, 1.26k op/s 2024-03-28T18:48:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.445+0000 7f50d5052640 1 -- 172.21.15.145:0/2409533832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d010d9c0 msgr2=0x7f50d010de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.445+0000 7f50d5052640 1 --2- 172.21.15.145:0/2409533832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d010d9c0 0x7f50d010de40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009900 tx=0x7f50c002bc60 comp rx=0 tx=0).stop 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.451+0000 7f50d5052640 1 -- 172.21.15.145:0/2409533832 shutdown_connections 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.451+0000 7f50d5052640 1 --2- 172.21.15.145:0/2409533832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d010d9c0 0x7f50d010de40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.451+0000 7f50d5052640 1 --2- 172.21.15.145:0/2409533832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010b010 0x7f50d010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.451+0000 7f50d5052640 1 --2- 172.21.15.145:0/2409533832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d01086a0 0x7f50d010aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.451+0000 7f50d5052640 1 -- 172.21.15.145:0/2409533832 >> 172.21.15.145:0/2409533832 conn(0x7f50d006e6e0 msgr2=0x7f50d0070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:01.454 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.454+0000 7f50d5052640 1 -- 172.21.15.145:0/2409533832 shutdown_connections 2024-03-28T18:48:01.455 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.454+0000 7f50d5052640 1 -- 172.21.15.145:0/2409533832 wait complete. 2024-03-28T18:48:01.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:01 smithi164 ceph-mon[36248]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 739 MiB/s wr, 1.26k op/s 2024-03-28T18:48:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.455+0000 7f50d5052640 1 Processor -- start 2024-03-28T18:48:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.463+0000 7f50d5052640 1 -- start start 2024-03-28T18:48:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d010b010 0x7f50d0119d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010d9c0 0x7f50d0117e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d0110820 con 0x7f50d01086a0 2024-03-28T18:48:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d01106a0 con 0x7f50d010b010 2024-03-28T18:48:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.464+0000 7f50d5052640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d01109a0 con 0x7f50d010d9c0 2024-03-28T18:48:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.470+0000 7f50ced9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.470+0000 7f50ced9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39618/0 (socket says 172.21.15.145:39618) 2024-03-28T18:48:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.470+0000 7f50ced9d640 1 -- 172.21.15.145:0/1958805399 learned_addr learned my addr 172.21.15.145:0/1958805399 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.470+0000 7f50cf59e640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010d9c0 0x7f50d0117e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.470+0000 7f50ce59c640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d010b010 0x7f50d0119d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.471+0000 7f50ced9d640 1 -- 172.21.15.145:0/1958805399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010d9c0 msgr2=0x7f50d0117e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.471+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010d9c0 0x7f50d0117e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.471+0000 7f50ced9d640 1 -- 172.21.15.145:0/1958805399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d010b010 msgr2=0x7f50d0119d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.471+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d010b010 0x7f50d0119d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.471+0000 7f50ced9d640 1 -- 172.21.15.145:0/1958805399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d01185c0 con 0x7f50d01086a0 2024-03-28T18:48:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.476+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f50c800ba50 tx=0x7f50c800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.479+0000 7f50b7fff640 1 -- 172.21.15.145:0/1958805399 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8019070 con 0x7f50d01086a0 2024-03-28T18:48:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.479+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d01188b0 con 0x7f50d01086a0 2024-03-28T18:48:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.479+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d01b7480 con 0x7f50d01086a0 2024-03-28T18:48:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.479+0000 7f50b7fff640 1 -- 172.21.15.145:0/1958805399 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c800cbe0 con 0x7f50d01086a0 2024-03-28T18:48:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.480+0000 7f50b7fff640 1 -- 172.21.15.145:0/1958805399 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8007660 con 0x7f50d01086a0 2024-03-28T18:48:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.485+0000 7f50b7fff640 1 -- 172.21.15.145:0/1958805399 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f50c801c070 con 0x7f50d01086a0 2024-03-28T18:48:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.486+0000 7f50b7fff640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 0x7f50b80a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.487+0000 7f50b7fff640 1 -- 172.21.15.145:0/1958805399 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f50c80cf450 con 0x7f50d01086a0 2024-03-28T18:48:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.491+0000 7f50ce59c640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 0x7f50b80a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:01 smithi145 ceph-mon[32151]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 739 MiB/s wr, 1.26k op/s 2024-03-28T18:48:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.549+0000 7f50ce59c640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 0x7f50b80a4fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f50d01195c0 tx=0x7f50c4005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 Processor -- start 2024-03-28T18:48:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 -- start start 2024-03-28T18:48:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a8fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 0x5622b1a94e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a97910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b19f4b10 con 0x5622b1a95480 2024-03-28T18:48:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b199d900 con 0x5622b1a8f920 2024-03-28T18:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.550+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b19f0bb0 con 0x5622b1a90310 2024-03-28T18:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.551+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 0x5622b1a94e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.551+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 0x5622b1a94e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:35182/0 (socket says 172.21.15.145:35182) 2024-03-28T18:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.551+0000 7f50ce59c640 1 -- 172.21.15.145:0/1696189467 learned_addr learned my addr 172.21.15.145:0/1696189467 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.552+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a8fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.555+0000 7f50ced9d640 1 -- 172.21.15.145:0/1696189467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 msgr2=0x5622b1a94e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.555+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 0x5622b1a94e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.555+0000 7f50ced9d640 1 -- 172.21.15.145:0/1696189467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 msgr2=0x5622b1a97910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.555+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a97910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.555+0000 7f50ced9d640 1 -- 172.21.15.145:0/1696189467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622b1709210 con 0x5622b1a8f920 2024-03-28T18:48:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.556+0000 7f50ced9d640 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a8fd40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50c80ccb40 tx=0x7f50c8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be742d0380a09e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.558+0000 7f50b57fa640 1 -- 172.21.15.145:0/1696189467 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80a4180 con 0x5622b1a8f920 2024-03-28T18:48:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.558+0000 7f50b57fa640 1 -- 172.21.15.145:0/1696189467 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c8099ce0 con 0x5622b1a8f920 2024-03-28T18:48:01.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.558+0000 7f50b57fa640 1 -- 172.21.15.145:0/1696189467 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80a3f50 con 0x5622b1a8f920 2024-03-28T18:48:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.560+0000 7f50d7c3e740 1 -- 172.21.15.145:0/1696189467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 msgr2=0x5622b1a8fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.560+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a8fd40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50c80ccb40 tx=0x7f50c8004a40 comp rx=0 tx=0).stop 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.563+0000 7f50d7c3e740 1 -- 172.21.15.145:0/1696189467 shutdown_connections 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.563+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a97910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.563+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a90310 0x5622b1a94e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.563+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/1696189467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a8fd40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.563+0000 7f50d7c3e740 1 -- 172.21.15.145:0/1696189467 >> 172.21.15.145:0/1696189467 conn(0x5622b1a8ae90 msgr2=0x5622b1a8b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.565+0000 7f50d7c3e740 1 -- 172.21.15.145:0/1696189467 shutdown_connections 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.565+0000 7f50d7c3e740 1 -- 172.21.15.145:0/1696189467 wait complete. 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.565+0000 7f50d7c3e740 1 -- start start 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.565+0000 7f50d7c3e740 1 Processor -- start 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a8f920 0x5622b1a99c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b1a98430 con 0x5622b1a95480 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b19b0d50 con 0x5622b1a90310 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622b1a985b0 con 0x5622b1a8f920 2024-03-28T18:48:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50d7c3e740 10 client.0 _ll_register_callbacks cb 0x5622b19b0720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.566+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:36166/0 (socket says 172.21.15.145:36166) 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 -- 172.21.15.145:0/869082638 learned_addr learned my addr 172.21.15.145:0/869082638 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ced9d640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a8f920 0x5622b1a99c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 -- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a8f920 msgr2=0x5622b1a99c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a8f920 0x5622b1a99c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 -- 172.21.15.145:0/869082638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 msgr2=0x5622b1aa1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.567+0000 7f50ce59c640 1 -- 172.21.15.145:0/869082638 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622b17d5620 con 0x5622b1a90310 2024-03-28T18:48:01.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.570+0000 7f50ce59c640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50c400f9a0 tx=0x7f50c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a478854b373515e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.570+0000 7f50b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.571+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400fe60 con 0x5622b1a90310 2024-03-28T18:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.571+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50c8099ce0 con 0x5622b1a90310 2024-03-28T18:48:01.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.571+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f50c401cd50 con 0x5622b1a90310 2024-03-28T18:48:01.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.571+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c4030050 con 0x5622b1a90310 2024-03-28T18:48:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.573+0000 7f50d7c3e740 10 client.25778 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.573+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5622b1aa4610 con 0x5622b1a90310 2024-03-28T18:48:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.574+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c8099ce0 con 0x5622b1a90310 2024-03-28T18:48:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.574+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f50c40256d0 con 0x5622b1a90310 2024-03-28T18:48:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 10 client.25778 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 10 client.25778 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 20 client.25778 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1aa6140 0x5622b1aa85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 4 client.25778 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.575+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622b1a02050 con 0x5622b1aa6140 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.576+0000 7f50ced9d640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1aa6140 0x5622b1aa85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.576+0000 7f50b2ff5640 10 client.25778 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.576+0000 7f50ced9d640 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1aa6140 0x5622b1aa85a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f002266e7a5fe68 server_cookie=11fd33ff44cd1a76 in_seq=0 out_seq=0 2024-03-28T18:48:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.578+0000 7f50b2ff5640 1 -- 172.21.15.145:0/869082638 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622b1a02050 con 0x5622b1aa6140 2024-03-28T18:48:01.579 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.578+0000 7f50b2ff5640 10 client.25778 handle_command_reply: tid=0 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.579+0000 7f50d7c3e740 1 client.25778 shutdown 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 msgr2=0x5622b1a9a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50c400f9a0 tx=0x7f50c40046c0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 shutdown_connections 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1aa6140 0x5622b1aa85a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a90310 0x5622b1a9a1c0 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50c400f9a0 tx=0x7f50c40046c0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/869082638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a8f920 0x5622b1a99c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 >> 172.21.15.145:0/869082638 conn(0x5622b1a8ae90 msgr2=0x5622b1a8a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.580+0000 7f50b2ff5640 0 client.25778 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.581+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 shutdown_connections 2024-03-28T18:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.581+0000 7f50d7c3e740 1 -- 172.21.15.145:0/869082638 wait complete. 2024-03-28T18:48:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.581+0000 7f50d7c3e740 20 client.25778 trim_cache size 0 max 16384 2024-03-28T18:48:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.610+0000 7f50d7c3e740 1 Processor -- start 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 -- start start 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a98fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc00fb40 con 0x5622b1a95480 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0114c0 con 0x5622b1a8f920 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.614+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0101a0 con 0x5622b1a91b70 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:36174/0 (socket says 172.21.15.145:36174) 2024-03-28T18:48:01.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 -- 172.21.15.145:0/2456820532 learned_addr learned my addr 172.21.15.145:0/2456820532 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 -- 172.21.15.145:0/2456820532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 msgr2=0x5622b1a98fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a98fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 -- 172.21.15.145:0/2456820532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 msgr2=0x5622b1aa11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 -- 172.21.15.145:0/2456820532 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622b1a8fd00 con 0x5622b1a8f920 2024-03-28T18:48:01.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.615+0000 7f50ced9d640 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50c8014e60 tx=0x7f50c8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b12a8e92fd453269 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.619+0000 7f50b4ff9640 1 -- 172.21.15.145:0/2456820532 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8098050 con 0x5622b1a8f920 2024-03-28T18:48:01.619 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.619+0000 7f50b4ff9640 1 -- 172.21.15.145:0/2456820532 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c80af070 con 0x5622b1a8f920 2024-03-28T18:48:01.620 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.620+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2456820532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 msgr2=0x5622b1a91630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:01.620 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.620+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50c8014e60 tx=0x7f50c8000f40 comp rx=0 tx=0).stop 2024-03-28T18:48:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.622+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2456820532 shutdown_connections 2024-03-28T18:48:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.622+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a98fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.622+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1aa11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.622+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2456820532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a91630 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.622+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2456820532 >> 172.21.15.145:0/2456820532 conn(0x5622b1a8ae90 msgr2=0x5622b1a8b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.623+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2456820532 shutdown_connections 2024-03-28T18:48:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.623+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2456820532 wait complete. 2024-03-28T18:48:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.623+0000 7f50d7c3e740 1 -- start start 2024-03-28T18:48:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.623+0000 7f50d7c3e740 1 Processor -- start 2024-03-28T18:48:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a9e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a9fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc010e60 con 0x5622b1a95480 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc010800 con 0x5622b1a8f920 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc00dd00 con 0x5622b1a91b70 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50d7c3e740 10 client.0 _ll_register_callbacks cb 0x7f50c400ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50ce59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:35212/0 (socket says 172.21.15.145:35212) 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50ce59c640 1 -- 172.21.15.145:0/2957482279 learned_addr learned my addr 172.21.15.145:0/2957482279 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.624+0000 7f50cf59e640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a9fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.627 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.627+0000 7f50ce59c640 1 -- 172.21.15.145:0/2957482279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 msgr2=0x5622b1a9e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.627 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.627+0000 7f50ce59c640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a9e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.627 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.627+0000 7f50ce59c640 1 -- 172.21.15.145:0/2957482279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 msgr2=0x5622b1a9fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:01.627 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.627+0000 7f50ce59c640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a9fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:01.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.627+0000 7f50ce59c640 1 -- 172.21.15.145:0/2957482279 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622b1aa03e0 con 0x5622b1a91b70 2024-03-28T18:48:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.630+0000 7f50ce59c640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f50c4012040 tx=0x7f50c401ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d8cafd6c8e320a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:01.630 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.630+0000 7f50b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c403a020 con 0x5622b1a91b70 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50c80af070 con 0x5622b1a91b70 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f50c4025800 con 0x5622b1a91b70 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c400fe60 con 0x5622b1a91b70 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50d7c3e740 10 client.25858 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:01.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5622b1b01a70 con 0x5622b1a91b70 2024-03-28T18:48:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.631+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c80af070 con 0x5622b1a91b70 2024-03-28T18:48:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f50c4025ce0 con 0x5622b1a91b70 2024-03-28T18:48:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 10 client.25858 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:01.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 10 client.25858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 20 client.25858 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1b023f0 0x5622b1b04810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 4 client.25858 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5622b19fc3a0 con 0x5622b1b023f0 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.632+0000 7f50ced9d640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1b023f0 0x5622b1b04810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.633+0000 7f50ced9d640 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1b023f0 0x5622b1b04810 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8af76ffeca0d4c1 server_cookie=1828cb4a5bd23d78 in_seq=0 out_seq=0 2024-03-28T18:48:01.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:01.633+0000 7f50b2ff5640 10 client.25858 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:03.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:03 smithi192 ceph-mon[36589]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 654 MiB/s wr, 1.22k op/s 2024-03-28T18:48:03.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:03 smithi164 ceph-mon[36248]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 654 MiB/s wr, 1.22k op/s 2024-03-28T18:48:03.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:03 smithi145 ceph-mon[32151]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 654 MiB/s wr, 1.22k op/s 2024-03-28T18:48:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.314+0000 7f50b2ff5640 1 -- 172.21.15.145:0/2957482279 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5622b19fc3a0 con 0x5622b1b023f0 2024-03-28T18:48:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.314+0000 7f50b2ff5640 10 client.25858 handle_command_reply: tid=0 2024-03-28T18:48:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.314+0000 7f50d7c3e740 1 client.25858 shutdown 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.314+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 msgr2=0x5622b1a9ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.314+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f50c4012040 tx=0x7f50c401ce00 comp rx=0 tx=0).stop 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 shutdown_connections 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5622b1b023f0 0x5622b1b04810 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5622b1a95480 0x5622b1a9fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5622b1a91b70 0x5622b1a9ee50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 --2- 172.21.15.145:0/2957482279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5622b1a8f920 0x5622b1a9e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 >> 172.21.15.145:0/2957482279 conn(0x5622b1a8ae90 msgr2=0x5622b1a98fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50b2ff5640 0 client.25858 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 shutdown_connections 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 1 -- 172.21.15.145:0/2957482279 wait complete. 2024-03-28T18:48:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.315+0000 7f50d7c3e740 20 client.25858 trim_cache size 0 max 16384 2024-03-28T18:48:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:48:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:48:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:48:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:48:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.316+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 msgr2=0x7f50b80a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.316+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 0x7f50b80a4fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f50d01195c0 tx=0x7f50c4005c60 comp rx=0 tx=0).stop 2024-03-28T18:48:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.317+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 msgr2=0x7f50d0119850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.317+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f50c800ba50 tx=0x7f50c800bf10 comp rx=0 tx=0).stop 2024-03-28T18:48:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 shutdown_connections 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f50b80a2ae0 0x7f50b80a4fa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50d010d9c0 0x7f50d0117e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d010b010 0x7f50d0119d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 --2- 172.21.15.145:0/1958805399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50d01086a0 0x7f50d0119850 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 >> 172.21.15.145:0/1958805399 conn(0x7f50d006e6e0 msgr2=0x7f50d0108d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 shutdown_connections 2024-03-28T18:48:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:04.319+0000 7f50d5052640 1 -- 172.21.15.145:0/1958805399 wait complete. 2024-03-28T18:48:05.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:05 smithi192 ceph-mon[36589]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 738 MiB/s wr, 1.57k op/s 2024-03-28T18:48:05.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:05 smithi164 ceph-mon[36248]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 738 MiB/s wr, 1.57k op/s 2024-03-28T18:48:05.547 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:05 smithi145 ceph-mon[32151]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 738 MiB/s wr, 1.57k op/s 2024-03-28T18:48:07.348 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:07 smithi145 ceph-mon[32151]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 625 MiB/s wr, 1.46k op/s 2024-03-28T18:48:07.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:07 smithi192 ceph-mon[36589]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 625 MiB/s wr, 1.46k op/s 2024-03-28T18:48:07.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:07 smithi164 ceph-mon[36248]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 625 MiB/s wr, 1.46k op/s 2024-03-28T18:48:09.411 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:09 smithi192 ceph-mon[36589]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 646 MiB/s wr, 1.76k op/s 2024-03-28T18:48:09.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:09 smithi164 ceph-mon[36248]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 646 MiB/s wr, 1.76k op/s 2024-03-28T18:48:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:09 smithi145 ceph-mon[32151]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 646 MiB/s wr, 1.76k op/s 2024-03-28T18:48:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:11 smithi192 ceph-mon[36589]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 678 MiB/s wr, 2.18k op/s 2024-03-28T18:48:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:11 smithi164 ceph-mon[36248]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 678 MiB/s wr, 2.18k op/s 2024-03-28T18:48:11.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:11 smithi145 ceph-mon[32151]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 678 MiB/s wr, 2.18k op/s 2024-03-28T18:48:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:13 smithi192 ceph-mon[36589]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 585 MiB/s wr, 2.12k op/s 2024-03-28T18:48:13.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:13 smithi164 ceph-mon[36248]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 585 MiB/s wr, 2.12k op/s 2024-03-28T18:48:13.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:13 smithi145 ceph-mon[32151]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 585 MiB/s wr, 2.12k op/s 2024-03-28T18:48:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:15 smithi192 ceph-mon[36589]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 635 MiB/s wr, 2.18k op/s 2024-03-28T18:48:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:15 smithi164 ceph-mon[36248]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 635 MiB/s wr, 2.18k op/s 2024-03-28T18:48:15.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:15 smithi145 ceph-mon[32151]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 635 MiB/s wr, 2.18k op/s 2024-03-28T18:48:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:17 smithi192 ceph-mon[36589]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 570 MiB/s wr, 1.86k op/s 2024-03-28T18:48:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:17 smithi164 ceph-mon[36248]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 570 MiB/s wr, 1.86k op/s 2024-03-28T18:48:17.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:17 smithi145 ceph-mon[32151]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 570 MiB/s wr, 1.86k op/s 2024-03-28T18:48:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.617+0000 7fd9e16f8640 1 -- 172.21.15.145:0/1281521893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 msgr2=0x7fd9dc10e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.617+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/1281521893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc10e060 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009900 tx=0x7fd9d002bc80 comp rx=0 tx=0).stop 2024-03-28T18:48:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.622+0000 7fd9e16f8640 1 -- 172.21.15.145:0/1281521893 shutdown_connections 2024-03-28T18:48:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.622+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/1281521893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc10e060 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.622+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/1281521893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9dc0732d0 0x7fd9dc108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.622+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/1281521893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc072970 0x7fd9dc072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.622+0000 7fd9e16f8640 1 -- 172.21.15.145:0/1281521893 >> 172.21.15.145:0/1281521893 conn(0x7fd9dc06e510 msgr2=0x7fd9dc070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.624+0000 7fd9e16f8640 1 -- 172.21.15.145:0/1281521893 shutdown_connections 2024-03-28T18:48:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.627+0000 7fd9e16f8640 1 -- 172.21.15.145:0/1281521893 wait complete. 2024-03-28T18:48:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.630+0000 7fd9e16f8640 1 Processor -- start 2024-03-28T18:48:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.630+0000 7fd9e16f8640 1 -- start start 2024-03-28T18:48:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.631+0000 7fd9e16f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9dc072970 0x7fd9dc1a6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.631+0000 7fd9e16f8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc0732d0 0x7fd9dc1a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.631+0000 7fd9e16f8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.632+0000 7fd9e16f8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc11c050 con 0x7fd9dc1093d0 2024-03-28T18:48:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.632+0000 7fd9e16f8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc11bed0 con 0x7fd9dc072970 2024-03-28T18:48:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.632+0000 7fd9e16f8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc11c1d0 con 0x7fd9dc0732d0 2024-03-28T18:48:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.635+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.635+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:55852/0 (socket says 172.21.15.145:55852) 2024-03-28T18:48:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.635+0000 7fd9db7fe640 1 -- 172.21.15.145:0/2848898975 learned_addr learned my addr 172.21.15.145:0/2848898975 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.636+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc0732d0 0x7fd9dc1a6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.637+0000 7fd9db7fe640 1 -- 172.21.15.145:0/2848898975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc0732d0 msgr2=0x7fd9dc1a6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.637+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc0732d0 0x7fd9dc1a6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.637+0000 7fd9db7fe640 1 -- 172.21.15.145:0/2848898975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9dc072970 msgr2=0x7fd9dc1a6690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:48:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.637+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9dc072970 0x7fd9dc1a6690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.637+0000 7fd9db7fe640 1 -- 172.21.15.145:0/2848898975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc1ae2b0 con 0x7fd9dc1093d0 2024-03-28T18:48:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.638+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009850 tx=0x7fd9d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.639+0000 7fd9c3fff640 1 -- 172.21.15.145:0/2848898975 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0043020 con 0x7fd9dc1093d0 2024-03-28T18:48:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.639+0000 7fd9c3fff640 1 -- 172.21.15.145:0/2848898975 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9d0004480 con 0x7fd9dc1093d0 2024-03-28T18:48:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.639+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc1ae540 con 0x7fd9dc1093d0 2024-03-28T18:48:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.639+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc1aea90 con 0x7fd9dc1093d0 2024-03-28T18:48:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.643+0000 7fd9e43c2740 1 Processor -- start 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.643+0000 7fd9e43c2740 1 -- start start 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9c3fff640 1 -- 172.21.15.145:0/2848898975 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0003df0 con 0x7fd9dc1093d0 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd61510 0x55dc5dd66010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd66680 0x55dc5dd68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dcc5b10 con 0x55dc5dd66680 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dc6e900 con 0x55dc5dd60b20 2024-03-28T18:48:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dcc1bb0 con 0x55dc5dd61510 2024-03-28T18:48:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9da7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9da7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:39276/0 (socket says 172.21.15.145:39276) 2024-03-28T18:48:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.644+0000 7fd9da7fc640 1 -- 172.21.15.145:0/1953923888 learned_addr learned my addr 172.21.15.145:0/1953923888 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.645+0000 7fd9da7fc640 1 -- 172.21.15.145:0/1953923888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd61510 msgr2=0x55dc5dd66010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:48:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.645+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd61510 0x55dc5dd66010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.645+0000 7fd9da7fc640 1 -- 172.21.15.145:0/1953923888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd66680 msgr2=0x55dc5dd68b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.645+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd66680 0x55dc5dd68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.652+0000 7fd9c3fff640 1 -- 172.21.15.145:0/2848898975 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd9d0046020 con 0x7fd9dc1093d0 2024-03-28T18:48:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.652+0000 7fd9da7fc640 1 -- 172.21.15.145:0/1953923888 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc5d9da210 con 0x55dc5dd60b20 2024-03-28T18:48:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.653+0000 7fd9c3fff640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 0x7fd9c40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.653+0000 7fd9c3fff640 1 -- 172.21.15.145:0/2848898975 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fd9d00f2180 con 0x7fd9dc1093d0 2024-03-28T18:48:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.654+0000 7fd9daffd640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 0x7fd9c40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.655+0000 7fd9daffd640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 0x7fd9c40a4e40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55dc5dd61910 tx=0x7fd9cc006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.656+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400afa0 tx=0x7fd9d4011870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae6ddbab232cc08a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.660+0000 7fd9c17fa640 1 -- 172.21.15.145:0/1953923888 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4004d20 con 0x55dc5dd60b20 2024-03-28T18:48:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.660+0000 7fd9c17fa640 1 -- 172.21.15.145:0/1953923888 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9d4016030 con 0x55dc5dd60b20 2024-03-28T18:48:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.660+0000 7fd9c17fa640 1 -- 172.21.15.145:0/1953923888 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4005720 con 0x55dc5dd60b20 2024-03-28T18:48:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 -- 172.21.15.145:0/1953923888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 msgr2=0x55dc5dd60f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400afa0 tx=0x7fd9d4011870 comp rx=0 tx=0).stop 2024-03-28T18:48:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 -- 172.21.15.145:0/1953923888 shutdown_connections 2024-03-28T18:48:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd66680 0x55dc5dd68b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd61510 0x55dc5dd66010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/1953923888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd60b20 0x55dc5dd60f40 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400afa0 tx=0x7fd9d4011870 comp rx=0 tx=0).stop 2024-03-28T18:48:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.662+0000 7fd9e43c2740 1 -- 172.21.15.145:0/1953923888 >> 172.21.15.145:0/1953923888 conn(0x55dc5dd5c090 msgr2=0x55dc5dd5c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.663+0000 7fd9e43c2740 1 -- 172.21.15.145:0/1953923888 shutdown_connections 2024-03-28T18:48:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.665+0000 7fd9e43c2740 1 -- 172.21.15.145:0/1953923888 wait complete. 2024-03-28T18:48:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.665+0000 7fd9e43c2740 1 -- start start 2024-03-28T18:48:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.666+0000 7fd9e43c2740 1 Processor -- start 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd7c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd81640 0x55dc5dd83a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dd7a640 con 0x55dc5dd61510 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dc81d50 con 0x55dc5dd66680 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc5dd7a7c0 con 0x55dc5dd81640 2024-03-28T18:48:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.670+0000 7fd9e43c2740 10 client.0 _ll_register_callbacks cb 0x55dc5dc81720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.672+0000 7fd9daffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.672+0000 7fd9daffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:55880/0 (socket says 172.21.15.145:55880) 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.672+0000 7fd9daffd640 1 -- 172.21.15.145:0/389138416 learned_addr learned my addr 172.21.15.145:0/389138416 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 -- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd81640 msgr2=0x55dc5dd83a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd81640 0x55dc5dd83a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 -- 172.21.15.145:0/389138416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 msgr2=0x55dc5dd7c680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd7c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 -- 172.21.15.145:0/389138416 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc5daa6620 con 0x55dc5dd61510 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9daffd640 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc013f40 tx=0x7fd9cc0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=386f25a63065d2d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9beff5640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.673+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc01d5f0 con 0x55dc5dd61510 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.674+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9d4016030 con 0x55dc5dd61510 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.674+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fd9cc01d790 con 0x55dc5dd61510 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.674+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9cc02b050 con 0x55dc5dd61510 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.674+0000 7fd9e43c2740 10 client.15987 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.674+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55dc5dd86e40 con 0x55dc5dd61510 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.675+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9d4016030 con 0x55dc5dd61510 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.675+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd9cc012070 con 0x55dc5dd61510 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 10 client.15987 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 20 client.15987 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dd88530 0x55dc5dd8a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 4 client.15987 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.678+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc5dcd3050 con 0x55dc5dd88530 2024-03-28T18:48:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.680+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dd88530 0x55dc5dd8a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.685+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dd88530 0x55dc5dd8a9d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=189e7cb5b7a18984 server_cookie=a808f9f8aa8524e2 in_seq=0 out_seq=0 2024-03-28T18:48:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.685+0000 7fd9beff5640 10 client.15987 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.689+0000 7fd9beff5640 1 -- 172.21.15.145:0/389138416 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dc5dcd3050 con 0x55dc5dd88530 2024-03-28T18:48:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.689+0000 7fd9beff5640 10 client.15987 handle_command_reply: tid=0 2024-03-28T18:48:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.690+0000 7fd9e43c2740 1 client.15987 shutdown 2024-03-28T18:48:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 msgr2=0x55dc5dd7c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc013f40 tx=0x7fd9cc0046e0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 shutdown_connections 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dd88530 0x55dc5dd8a9d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd81640 0x55dc5dd83a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd7c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/389138416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd61510 0x55dc5dd7c140 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc013f40 tx=0x7fd9cc0046e0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.694+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 >> 172.21.15.145:0/389138416 conn(0x55dc5dd5c090 msgr2=0x55dc5dd5b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.695+0000 7fd9beff5640 0 client.15987 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.696+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 shutdown_connections 2024-03-28T18:48:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.697+0000 7fd9e43c2740 1 -- 172.21.15.145:0/389138416 wait complete. 2024-03-28T18:48:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.697+0000 7fd9e43c2740 20 client.15987 trim_cache size 0 max 16384 2024-03-28T18:48:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.734+0000 7fd9e43c2740 1 Processor -- start 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 -- start start 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd5c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd7c290 0x55dc5dd5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c800fb40 con 0x55dc5dd81640 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80114c0 con 0x55dc5dd66680 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80101a0 con 0x55dc5dd7c290 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9da7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9da7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:55882/0 (socket says 172.21.15.145:55882) 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.736+0000 7fd9da7fc640 1 -- 172.21.15.145:0/3102125069 learned_addr learned my addr 172.21.15.145:0/3102125069 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.738+0000 7fd9da7fc640 1 -- 172.21.15.145:0/3102125069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd7c290 msgr2=0x55dc5dd5aa00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.738+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd7c290 0x55dc5dd5aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.738+0000 7fd9da7fc640 1 -- 172.21.15.145:0/3102125069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 msgr2=0x55dc5dd5c470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:48:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.738+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd5c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.738+0000 7fd9da7fc640 1 -- 172.21.15.145:0/3102125069 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc5dd5b9e0 con 0x55dc5dd81640 2024-03-28T18:48:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.739+0000 7fd9da7fc640 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400af70 tx=0x7fd9d4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9afd457667f9793 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.745+0000 7fd9c0ff9640 1 -- 172.21.15.145:0/3102125069 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4004d20 con 0x55dc5dd81640 2024-03-28T18:48:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.745+0000 7fd9c0ff9640 1 -- 172.21.15.145:0/3102125069 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9d4016040 con 0x55dc5dd81640 2024-03-28T18:48:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.745+0000 7fd9c0ff9640 1 -- 172.21.15.145:0/3102125069 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4028a30 con 0x55dc5dd81640 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3102125069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 msgr2=0x55dc5dd7bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400af70 tx=0x7fd9d4000f40 comp rx=0 tx=0).stop 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3102125069 shutdown_connections 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd7c290 0x55dc5dd5aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd81640 0x55dc5dd7bd50 secure :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400af70 tx=0x7fd9d4000f40 comp rx=0 tx=0).stop 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3102125069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd5c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.747+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3102125069 >> 172.21.15.145:0/3102125069 conn(0x55dc5dd5c090 msgr2=0x55dc5dd7abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:17.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.754+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3102125069 shutdown_connections 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.757+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3102125069 wait complete. 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 -- start start 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 Processor -- start 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd9a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd7c290 0x55dc5dd98f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c8010e60 con 0x55dc5dd7c290 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c8010800 con 0x55dc5dd66680 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c800dd00 con 0x55dc5dd99480 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.758+0000 7fd9e43c2740 10 client.0 _ll_register_callbacks cb 0x7fd9cc013bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.759+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.759+0000 7fd9db7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57968/0 (socket says 172.21.15.145:57968) 2024-03-28T18:48:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.759+0000 7fd9db7fe640 1 -- 172.21.15.145:0/3761476768 learned_addr learned my addr 172.21.15.145:0/3761476768 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9daffd640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd9a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9db7fe640 1 -- 172.21.15.145:0/3761476768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 msgr2=0x55dc5dd9a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd9a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9db7fe640 1 -- 172.21.15.145:0/3761476768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd7c290 msgr2=0x55dc5dd98f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd7c290 0x55dc5dd98f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.760+0000 7fd9db7fe640 1 -- 172.21.15.145:0/3761476768 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc5dde3350 con 0x55dc5dd99480 2024-03-28T18:48:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.761+0000 7fd9db7fe640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0003a00 tx=0x7fd9d00be190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee2da20d33214534 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.761+0000 7fd9beff5640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:48:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.761+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00c28c0 con 0x55dc5dd99480 2024-03-28T18:48:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.761+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9d4016040 con 0x55dc5dd99480 2024-03-28T18:48:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.761+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fd9d00035f0 con 0x55dc5dd99480 2024-03-28T18:48:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.762+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9d00c6f10 con 0x55dc5dd99480 2024-03-28T18:48:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.763+0000 7fd9e43c2740 10 client.25864 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.763+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55dc5dde4660 con 0x55dc5dd99480 2024-03-28T18:48:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9d4016040 con 0x55dc5dd99480 2024-03-28T18:48:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd9d00bdbc0 con 0x55dc5dd99480 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 10 client.25864 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 10 client.25864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 20 client.25864 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dde5040 0x55dc5dde7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 4 client.25864 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:48:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.765+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dc5dccd3a0 con 0x55dc5dde5040 2024-03-28T18:48:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.770+0000 7fd9daffd640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dde5040 0x55dc5dde7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.774+0000 7fd9daffd640 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dde5040 0x55dc5dde7460 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e44c55250e3cd0 server_cookie=5a08e842956e8dac in_seq=0 out_seq=0 2024-03-28T18:48:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:17.776+0000 7fd9beff5640 10 client.25864 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:19 smithi192 ceph-mon[36589]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 571 MiB/s wr, 1.76k op/s 2024-03-28T18:48:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:19 smithi164 ceph-mon[36248]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 571 MiB/s wr, 1.76k op/s 2024-03-28T18:48:19.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:19 smithi145 ceph-mon[32151]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 571 MiB/s wr, 1.76k op/s 2024-03-28T18:48:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:21 smithi192 ceph-mon[36589]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 637 MiB/s wr, 1.65k op/s 2024-03-28T18:48:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:21 smithi164 ceph-mon[36248]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 637 MiB/s wr, 1.65k op/s 2024-03-28T18:48:21.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:21 smithi145 ceph-mon[32151]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 637 MiB/s wr, 1.65k op/s 2024-03-28T18:48:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.167+0000 7fd9beff5640 1 -- 172.21.15.145:0/3761476768 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dc5dccd3a0 con 0x55dc5dde5040 2024-03-28T18:48:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.167+0000 7fd9beff5640 10 client.25864 handle_command_reply: tid=0 2024-03-28T18:48:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.167+0000 7fd9e43c2740 1 client.25864 shutdown 2024-03-28T18:48:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 msgr2=0x55dc5dd99900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0003a00 tx=0x7fd9d00be190 comp rx=0 tx=0).stop 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 shutdown_connections 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55dc5dde5040 0x55dc5dde7460 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55dc5dd99480 0x55dc5dd99900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dc5dd7c290 0x55dc5dd98f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 --2- 172.21.15.145:0/3761476768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc5dd66680 0x55dc5dd9a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 >> 172.21.15.145:0/3761476768 conn(0x55dc5dd5c090 msgr2=0x55dc5dd5c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9beff5640 0 client.25864 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 shutdown_connections 2024-03-28T18:48:22.169 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 1 -- 172.21.15.145:0/3761476768 wait complete. 2024-03-28T18:48:22.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.168+0000 7fd9e43c2740 20 client.25864 trim_cache size 0 max 16384 2024-03-28T18:48:22.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:48:22.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:48:22.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:48:22.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 msgr2=0x7fd9c40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 0x7fd9c40a4e40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55dc5dd61910 tx=0x7fd9cc006bd0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 msgr2=0x7fd9dc1adc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009850 tx=0x7fd9d0004040 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 shutdown_connections 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9dc1093d0 0x7fd9dc1adc40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fd9c40a2980 0x7fd9c40a4e40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9dc0732d0 0x7fd9dc1a6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 --2- 172.21.15.145:0/2848898975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9dc072970 0x7fd9dc1a6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.169+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 >> 172.21.15.145:0/2848898975 conn(0x7fd9dc06e510 msgr2=0x7fd9dc070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.170+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 shutdown_connections 2024-03-28T18:48:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:22.170+0000 7fd9e16f8640 1 -- 172.21.15.145:0/2848898975 wait complete. 2024-03-28T18:48:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:23 smithi192 ceph-mon[36589]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 553 MiB/s wr, 1.11k op/s 2024-03-28T18:48:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:23 smithi164 ceph-mon[36248]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 553 MiB/s wr, 1.11k op/s 2024-03-28T18:48:23.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:23 smithi145 ceph-mon[32151]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 553 MiB/s wr, 1.11k op/s 2024-03-28T18:48:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:25 smithi192 ceph-mon[36589]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 681 MiB/s wr, 1.21k op/s 2024-03-28T18:48:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:25 smithi164 ceph-mon[36248]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 681 MiB/s wr, 1.21k op/s 2024-03-28T18:48:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:25 smithi145 ceph-mon[32151]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 681 MiB/s wr, 1.21k op/s 2024-03-28T18:48:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:27 smithi192 ceph-mon[36589]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 639 MiB/s wr, 1.15k op/s 2024-03-28T18:48:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:27 smithi164 ceph-mon[36248]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 639 MiB/s wr, 1.15k op/s 2024-03-28T18:48:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:27 smithi145 ceph-mon[32151]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 639 MiB/s wr, 1.15k op/s 2024-03-28T18:48:29.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:29 smithi192 ceph-mon[36589]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 600 MiB/s wr, 1.19k op/s 2024-03-28T18:48:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:29 smithi164 ceph-mon[36248]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 600 MiB/s wr, 1.19k op/s 2024-03-28T18:48:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:29 smithi145 ceph-mon[32151]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 600 MiB/s wr, 1.19k op/s 2024-03-28T18:48:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:31 smithi164 ceph-mon[36248]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 706 MiB/s wr, 1.80k op/s 2024-03-28T18:48:31.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:31 smithi145 ceph-mon[32151]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 706 MiB/s wr, 1.80k op/s 2024-03-28T18:48:31.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:31 smithi192 ceph-mon[36589]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 706 MiB/s wr, 1.80k op/s 2024-03-28T18:48:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:33 smithi164 ceph-mon[36248]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 594 MiB/s wr, 1.75k op/s 2024-03-28T18:48:33.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:33 smithi145 ceph-mon[32151]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 594 MiB/s wr, 1.75k op/s 2024-03-28T18:48:33.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:33 smithi192 ceph-mon[36589]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 594 MiB/s wr, 1.75k op/s 2024-03-28T18:48:35.424 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.424+0000 7efd4ffff640 1 -- 172.21.15.145:0/863506409 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48004d50 con 0x7efd5810d9c0 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.426+0000 7efd5f562640 1 -- 172.21.15.145:0/863506409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 msgr2=0x7efd5810de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.426+0000 7efd5f562640 1 --2- 172.21.15.145:0/863506409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd5810de40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efd4801a910 tx=0x7efd4803cc60 comp rx=0 tx=0).stop 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.428+0000 7efd5f562640 1 -- 172.21.15.145:0/863506409 shutdown_connections 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.428+0000 7efd5f562640 1 --2- 172.21.15.145:0/863506409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd5810de40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.428+0000 7efd5f562640 1 --2- 172.21.15.145:0/863506409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd5810b010 0x7efd5810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.428+0000 7efd5f562640 1 --2- 172.21.15.145:0/863506409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd581086a0 0x7efd5810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.428+0000 7efd5f562640 1 -- 172.21.15.145:0/863506409 >> 172.21.15.145:0/863506409 conn(0x7efd5806e6e0 msgr2=0x7efd58070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.433 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.432+0000 7efd5f562640 1 -- 172.21.15.145:0/863506409 shutdown_connections 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.434+0000 7efd5f562640 1 -- 172.21.15.145:0/863506409 wait complete. 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.434+0000 7efd5f562640 1 Processor -- start 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.434+0000 7efd5f562640 1 -- start start 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd581086a0 0x7efd5819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5810b010 0x7efd5819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581119b0 con 0x7efd5810b010 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58111830 con 0x7efd5810d9c0 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.435+0000 7efd5f562640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58111b30 con 0x7efd581086a0 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.436+0000 7efd5daff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.437+0000 7efd5daff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:46538/0 (socket says 172.21.15.145:46538) 2024-03-28T18:48:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.437+0000 7efd5daff640 1 -- 172.21.15.145:0/3413367802 learned_addr learned my addr 172.21.15.145:0/3413367802 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd581086a0 0x7efd5819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5daff640 1 -- 172.21.15.145:0/3413367802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd581086a0 msgr2=0x7efd5819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5daff640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd581086a0 0x7efd5819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5daff640 1 -- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5810b010 msgr2=0x7efd5819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5daff640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5810b010 0x7efd5819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.439 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.438+0000 7efd5daff640 1 -- 172.21.15.145:0/3413367802 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58079fa0 con 0x7efd5810d9c0 2024-03-28T18:48:35.440 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.439+0000 7efd5daff640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efd48005ab0 tx=0x7efd48004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.442+0000 7efd4e7fc640 1 -- 172.21.15.145:0/3413367802 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48052020 con 0x7efd5810d9c0 2024-03-28T18:48:35.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.442+0000 7efd4e7fc640 1 -- 172.21.15.145:0/3413367802 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd48004f30 con 0x7efd5810d9c0 2024-03-28T18:48:35.442 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.442+0000 7efd4e7fc640 1 -- 172.21.15.145:0/3413367802 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4804e480 con 0x7efd5810d9c0 2024-03-28T18:48:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.445+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5807a230 con 0x7efd5810d9c0 2024-03-28T18:48:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.445+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5807a780 con 0x7efd5810d9c0 2024-03-28T18:48:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.451+0000 7efd6222f740 1 Processor -- start 2024-03-28T18:48:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.451+0000 7efd6222f740 1 -- start start 2024-03-28T18:48:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e8dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.452 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e92e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e95910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49df2b10 con 0x558c49e8d920 2024-03-28T18:48:35.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49d9b900 con 0x558c49e8e310 2024-03-28T18:48:35.453 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.452+0000 7efd6222f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49deebb0 con 0x558c49e93480 2024-03-28T18:48:35.460 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.454+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e95910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.454+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e95910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56412/0 (socket says 172.21.15.145:56412) 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.454+0000 7efd5cafd640 1 -- 172.21.15.145:0/1763214516 learned_addr learned my addr 172.21.15.145:0/1763214516 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.454+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e8dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.455+0000 7efd4e7fc640 1 -- 172.21.15.145:0/3413367802 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efd48055020 con 0x7efd5810d9c0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.456+0000 7efd4e7fc640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 0x7efd440a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.456+0000 7efd5daff640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 0x7efd440a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5cafd640 1 -- 172.21.15.145:0/1763214516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 msgr2=0x558c49e95910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e95910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5cafd640 1 -- 172.21.15.145:0/1763214516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 msgr2=0x558c49e92e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e92e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5cafd640 1 -- 172.21.15.145:0/1763214516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c49b07210 con 0x558c49e8d920 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd5daff640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 0x7efd440a4f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efd480d1cd0 tx=0x7efd480c9c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.457+0000 7efd4e7fc640 1 -- 172.21.15.145:0/3413367802 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd48101ab0 con 0x7efd5810d9c0 2024-03-28T18:48:35.461 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.458+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e8dd40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd54013760 tx=0x7efd54005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7787ebe29b00d9d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:35 smithi164 ceph-mon[36248]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 661 MiB/s wr, 2.55k op/s 2024-03-28T18:48:35.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.460+0000 7efd437fe640 1 -- 172.21.15.145:0/1763214516 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54013da0 con 0x558c49e8d920 2024-03-28T18:48:35.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.461+0000 7efd437fe640 1 -- 172.21.15.145:0/1763214516 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd54017070 con 0x558c49e8d920 2024-03-28T18:48:35.462 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 -- 172.21.15.145:0/1763214516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 msgr2=0x558c49e8dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e8dd40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd54013760 tx=0x7efd54005ac0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 -- 172.21.15.145:0/1763214516 shutdown_connections 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e95910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e92e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 --2- 172.21.15.145:0/1763214516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e8dd40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.462+0000 7efd6222f740 1 -- 172.21.15.145:0/1763214516 >> 172.21.15.145:0/1763214516 conn(0x558c49e88e90 msgr2=0x558c49e89290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.473+0000 7efd6222f740 1 -- 172.21.15.145:0/1763214516 shutdown_connections 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.477+0000 7efd6222f740 1 -- 172.21.15.145:0/1763214516 wait complete. 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.478+0000 7efd6222f740 1 -- start start 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.478+0000 7efd6222f740 1 Processor -- start 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e97c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e98170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49e96480 con 0x558c49e8e310 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49daed50 con 0x558c49e93480 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c49e96600 con 0x558c49e8d920 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd6222f740 10 client.0 _ll_register_callbacks cb 0x558c49dae720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:46556/0 (socket says 172.21.15.145:46556) 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.479+0000 7efd5cafd640 1 -- 172.21.15.145:0/2564790408 learned_addr learned my addr 172.21.15.145:0/2564790408 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.480+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e98170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 msgr2=0x558c49e97c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e97c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2564790408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 msgr2=0x558c49e9f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2564790408 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c49bd3620 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.481+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e98170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x558c49e8e710 tx=0x7efd5000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41b6cacf88dad20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.482+0000 7efd40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.482+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50015070 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.482+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd50011840 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.482+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd54017070 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd5001f050 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 10 client.15996 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558c49ec4600 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd54017070 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efd500193b0 con 0x558c49e8e310 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 10 client.15996 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 20 client.15996 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:35.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49ec6160 0x558c49ec8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 4 client.15996 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.483+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c49e00050 con 0x558c49ec6160 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.484+0000 7efd5cafd640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49ec6160 0x558c49ec8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.485+0000 7efd40ff9640 10 client.15996 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.485+0000 7efd5cafd640 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49ec6160 0x558c49ec8580 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfb947c03b4910ff server_cookie=a9a489483bd44f4 in_seq=0 out_seq=0 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.489+0000 7efd40ff9640 1 -- 172.21.15.145:0/2564790408 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558c49e00050 con 0x558c49ec6160 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.489+0000 7efd40ff9640 10 client.15996 handle_command_reply: tid=0 2024-03-28T18:48:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.489+0000 7efd6222f740 1 client.15996 shutdown 2024-03-28T18:48:35.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.497+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 msgr2=0x558c49e98170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.498 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.497+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e98170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x558c49e8e710 tx=0x7efd5000c9b0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 shutdown_connections 2024-03-28T18:48:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49ec6160 0x558c49ec8580 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e98170 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 --2- 172.21.15.145:0/2564790408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e97c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.503+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 >> 172.21.15.145:0/2564790408 conn(0x558c49e88e90 msgr2=0x558c49e88020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.504+0000 7efd40ff9640 0 client.15996 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.508+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 shutdown_connections 2024-03-28T18:48:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.510+0000 7efd6222f740 1 -- 172.21.15.145:0/2564790408 wait complete. 2024-03-28T18:48:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.513+0000 7efd6222f740 20 client.15996 trim_cache size 0 max 16384 2024-03-28T18:48:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:35 smithi145 ceph-mon[32151]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 661 MiB/s wr, 2.55k op/s 2024-03-28T18:48:35.547 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 Processor -- start 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 -- start start 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e89270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e8f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581c17a0 con 0x558c49e8d920 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581c3120 con 0x558c49e8e310 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd6222f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581c1e00 con 0x558c49e93480 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd5d2fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd5d2fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:46572/0 (socket says 172.21.15.145:46572) 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.547+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2337235489 learned_addr learned my addr 172.21.15.145:0/2337235489 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5cafd640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e89270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5cafd640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e8f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2337235489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 msgr2=0x558c49e8f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e8f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2337235489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 msgr2=0x558c49e89270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e89270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5d2fe640 1 -- 172.21.15.145:0/2337235489 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c49e8f080 con 0x558c49e8e310 2024-03-28T18:48:35.549 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.548+0000 7efd5cafd640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e8f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:35.558 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.557+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd50019850 tx=0x7efd500073b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=941e2362bcd44718 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.559+0000 7efd42ffd640 1 -- 172.21.15.145:0/2337235489 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50015040 con 0x558c49e8e310 2024-03-28T18:48:35.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.559+0000 7efd42ffd640 1 -- 172.21.15.145:0/2337235489 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd500199c0 con 0x558c49e8e310 2024-03-28T18:48:35.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.559+0000 7efd42ffd640 1 -- 172.21.15.145:0/2337235489 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd500246a0 con 0x558c49e8e310 2024-03-28T18:48:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.561+0000 7efd6222f740 1 -- 172.21.15.145:0/2337235489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 msgr2=0x558c49e9f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.561+0000 7efd6222f740 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd50019850 tx=0x7efd500073b0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.563+0000 7efd6222f740 1 -- 172.21.15.145:0/2337235489 shutdown_connections 2024-03-28T18:48:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.563+0000 7efd6222f740 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e93480 0x558c49e8f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.563+0000 7efd6222f740 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e8e310 0x558c49e9f170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.563+0000 7efd6222f740 1 --2- 172.21.15.145:0/2337235489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8d920 0x558c49e89270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.563+0000 7efd6222f740 1 -- 172.21.15.145:0/2337235489 >> 172.21.15.145:0/2337235489 conn(0x558c49e88e90 msgr2=0x558c49e92bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.568+0000 7efd6222f740 1 -- 172.21.15.145:0/2337235489 shutdown_connections 2024-03-28T18:48:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.568+0000 7efd6222f740 1 -- 172.21.15.145:0/2337235489 wait complete. 2024-03-28T18:48:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.568+0000 7efd6222f740 1 -- start start 2024-03-28T18:48:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.568+0000 7efd6222f740 1 Processor -- start 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e9ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581c2ac0 con 0x558c49e8e310 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd581c2460 con 0x558c49e93480 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.570+0000 7efd6222f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58082df0 con 0x558c49e8d920 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.571+0000 7efd6222f740 10 client.0 _ll_register_callbacks cb 0x7efd50007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.571+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.571+0000 7efd5cafd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:56428/0 (socket says 172.21.15.145:56428) 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.571+0000 7efd5cafd640 1 -- 172.21.15.145:0/1708904806 learned_addr learned my addr 172.21.15.145:0/1708904806 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.571+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.572+0000 7efd5cafd640 1 -- 172.21.15.145:0/1708904806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 msgr2=0x558c49e9dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.572+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.572+0000 7efd5cafd640 1 -- 172.21.15.145:0/1708904806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 msgr2=0x558c49e9ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.572+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e9ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.572+0000 7efd5cafd640 1 -- 172.21.15.145:0/1708904806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c49e9e1b0 con 0x558c49e8d920 2024-03-28T18:48:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.574+0000 7efd5cafd640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd54005bc0 tx=0x7efd5400c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e11225cb58b4f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.574+0000 7efd40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:48:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54029700 con 0x558c49e8d920 2024-03-28T18:48:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd500199c0 con 0x558c49e8d920 2024-03-28T18:48:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efd540298a0 con 0x558c49e8d920 2024-03-28T18:48:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd54029aa0 con 0x558c49e8d920 2024-03-28T18:48:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd6222f740 10 client.25873 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.575+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558c49e9f4c0 con 0x558c49e8d920 2024-03-28T18:48:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd500199c0 con 0x558c49e8d920 2024-03-28T18:48:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efd5400c020 con 0x558c49e8d920 2024-03-28T18:48:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 10 client.25873 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 10 client.25873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 20 client.25873 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49f22110 0x558c49f24530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 4 client.25873 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.582+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558c49dfa3a0 con 0x558c49f22110 2024-03-28T18:48:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.583+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49f22110 0x558c49f24530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:35.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.587+0000 7efd40ff9640 10 client.25873 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:35.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.588+0000 7efd5d2fe640 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49f22110 0x558c49f24530 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b85858707ee74f8 server_cookie=f0118f9a61935a8c in_seq=0 out_seq=0 2024-03-28T18:48:35.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:35 smithi192 ceph-mon[36589]: pgmap v675: 129 pgs: 129 active+clean; 10 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 661 MiB/s wr, 2.55k op/s 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd40ff9640 1 -- 172.21.15.145:0/1708904806 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558c49dfa3a0 con 0x558c49f22110 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd40ff9640 10 client.25873 handle_command_reply: tid=0 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 client.25873 shutdown 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 msgr2=0x558c49e9c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd54005bc0 tx=0x7efd5400c970 comp rx=0 tx=0).stop 2024-03-28T18:48:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 shutdown_connections 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558c49f22110 0x558c49f24530 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558c49e93480 0x558c49e9dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558c49e8e310 0x558c49e9ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 --2- 172.21.15.145:0/1708904806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558c49e8d920 0x558c49e9c780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.930+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 >> 172.21.15.145:0/1708904806 conn(0x558c49e88e90 msgr2=0x558c49e97c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.931+0000 7efd40ff9640 0 client.25873 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.931+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 shutdown_connections 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.931+0000 7efd6222f740 1 -- 172.21.15.145:0/1708904806 wait complete. 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.931+0000 7efd6222f740 20 client.25873 trim_cache size 0 max 16384 2024-03-28T18:48:35.932 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 msgr2=0x7efd440a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 0x7efd440a4f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7efd480d1cd0 tx=0x7efd480c9c30 comp rx=0 tx=0).stop 2024-03-28T18:48:35.933 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 msgr2=0x7efd58079900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efd48005ab0 tx=0x7efd48004270 comp rx=0 tx=0).stop 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 shutdown_connections 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd5810d9c0 0x7efd58079900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5810b010 0x7efd5819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7efd440a2a90 0x7efd440a4f50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 --2- 172.21.15.145:0/3413367802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd581086a0 0x7efd5819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 >> 172.21.15.145:0/3413367802 conn(0x7efd5806e6e0 msgr2=0x7efd58108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 shutdown_connections 2024-03-28T18:48:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:35.932+0000 7efd5f562640 1 -- 172.21.15.145:0/3413367802 wait complete. 2024-03-28T18:48:37.459 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:37 smithi192 ceph-mon[36589]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 572 MiB/s wr, 2.94k op/s 2024-03-28T18:48:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:37 smithi164 ceph-mon[36248]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 572 MiB/s wr, 2.94k op/s 2024-03-28T18:48:37.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:37 smithi145 ceph-mon[32151]: pgmap v676: 129 pgs: 129 active+clean; 10 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 572 MiB/s wr, 2.94k op/s 2024-03-28T18:48:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:39 smithi164 ceph-mon[36248]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 512 MiB/s wr, 3.11k op/s 2024-03-28T18:48:39.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:39 smithi145 ceph-mon[32151]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 512 MiB/s wr, 3.11k op/s 2024-03-28T18:48:39.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:39 smithi192 ceph-mon[36589]: pgmap v677: 129 pgs: 129 active+clean; 10 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 512 MiB/s wr, 3.11k op/s 2024-03-28T18:48:41.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:41 smithi164 ceph-mon[36248]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 562 MiB/s wr, 3.93k op/s 2024-03-28T18:48:41.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:41 smithi145 ceph-mon[32151]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 562 MiB/s wr, 3.93k op/s 2024-03-28T18:48:41.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:41 smithi192 ceph-mon[36589]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s rd, 562 MiB/s wr, 3.93k op/s 2024-03-28T18:48:43.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:43 smithi164 ceph-mon[36248]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 442 MiB/s wr, 3.74k op/s 2024-03-28T18:48:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:43 smithi145 ceph-mon[32151]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 442 MiB/s wr, 3.74k op/s 2024-03-28T18:48:43.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:43 smithi192 ceph-mon[36589]: pgmap v679: 129 pgs: 129 active+clean; 10 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 442 MiB/s wr, 3.74k op/s 2024-03-28T18:48:45.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:45 smithi145 ceph-mon[32151]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 459 MiB/s wr, 3.95k op/s 2024-03-28T18:48:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:45 smithi192 ceph-mon[36589]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 459 MiB/s wr, 3.95k op/s 2024-03-28T18:48:45.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:45 smithi164 ceph-mon[36248]: pgmap v680: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 459 MiB/s wr, 3.95k op/s 2024-03-28T18:48:47.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:47 smithi145 ceph-mon[32151]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 490 MiB/s wr, 3.97k op/s 2024-03-28T18:48:47.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:47 smithi192 ceph-mon[36589]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 490 MiB/s wr, 3.97k op/s 2024-03-28T18:48:47.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:47 smithi164 ceph-mon[36248]: pgmap v681: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 490 MiB/s wr, 3.97k op/s 2024-03-28T18:48:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.163+0000 7f30bb310640 1 -- 172.21.15.145:0/1948136209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b40756e0 msgr2=0x7f30b4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.163+0000 7f30bb310640 1 --2- 172.21.15.145:0/1948136209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b40756e0 0x7f30b4073b30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009900 tx=0x7f30a402bc60 comp rx=0 tx=0).stop 2024-03-28T18:48:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.166+0000 7f30bb310640 1 -- 172.21.15.145:0/1948136209 shutdown_connections 2024-03-28T18:48:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.166+0000 7f30bb310640 1 --2- 172.21.15.145:0/1948136209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b410dcc0 0x7f30b41100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.166+0000 7f30bb310640 1 --2- 172.21.15.145:0/1948136209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b4074100 0x7f30b4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.166+0000 7f30bb310640 1 --2- 172.21.15.145:0/1948136209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b40756e0 0x7f30b4073b30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.166+0000 7f30bb310640 1 -- 172.21.15.145:0/1948136209 >> 172.21.15.145:0/1948136209 conn(0x7f30b40fcc60 msgr2=0x7f30b40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.170 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.170+0000 7f30bb310640 1 -- 172.21.15.145:0/1948136209 shutdown_connections 2024-03-28T18:48:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.171+0000 7f30bb310640 1 -- 172.21.15.145:0/1948136209 wait complete. 2024-03-28T18:48:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 Processor -- start 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 -- start start 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b4074100 0x7f30b410a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b410dcc0 0x7f30b41ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4113a10 con 0x7f30b40756e0 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4113890 con 0x7f30b410dcc0 2024-03-28T18:48:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.172+0000 7f30bb310640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4113b90 con 0x7f30b4074100 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.173+0000 7f30b88ab640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.173+0000 7f30b88ab640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43882/0 (socket says 172.21.15.145:43882) 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.173+0000 7f30b88ab640 1 -- 172.21.15.145:0/1223200343 learned_addr learned my addr 172.21.15.145:0/1223200343 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.173+0000 7f30b98ad640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b410dcc0 0x7f30b41ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.174+0000 7f30b88ab640 1 -- 172.21.15.145:0/1223200343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b4074100 msgr2=0x7f30b410a330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.174+0000 7f30b88ab640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b4074100 0x7f30b410a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.174+0000 7f30b88ab640 1 -- 172.21.15.145:0/1223200343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b410dcc0 msgr2=0x7f30b41ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.174+0000 7f30b88ab640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b410dcc0 0x7f30b41ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.174+0000 7f30b88ab640 1 -- 172.21.15.145:0/1223200343 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b410d9f0 con 0x7f30b40756e0 2024-03-28T18:48:49.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.175+0000 7f30b88ab640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c890 tx=0x7f30b000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.178+0000 7f30aa7fc640 1 -- 172.21.15.145:0/1223200343 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0004220 con 0x7f30b40756e0 2024-03-28T18:48:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.178+0000 7f30aa7fc640 1 -- 172.21.15.145:0/1223200343 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b00043c0 con 0x7f30b40756e0 2024-03-28T18:48:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.178+0000 7f30aa7fc640 1 -- 172.21.15.145:0/1223200343 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b00049c0 con 0x7f30b40756e0 2024-03-28T18:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.182+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b406b120 con 0x7f30b40756e0 2024-03-28T18:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.182+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b406b590 con 0x7f30b40756e0 2024-03-28T18:48:49.190 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.189+0000 7f30aa7fc640 1 -- 172.21.15.145:0/1223200343 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f30b0004b60 con 0x7f30b40756e0 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.190+0000 7f30bdfde740 1 Processor -- start 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.190+0000 7f30bdfde740 1 -- start start 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.190+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.190+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087278310 0x55f08727ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f08727f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30bdfde740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0871dcb10 con 0x55f087277920 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30bdfde740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f087185900 con 0x55f08727d480 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30bdfde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0871d8bb0 con 0x55f087278310 2024-03-28T18:48:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30b98ad640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30b98ad640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:43884/0 (socket says 172.21.15.145:43884) 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.191+0000 7f30b98ad640 1 -- 172.21.15.145:0/3701062912 learned_addr learned my addr 172.21.15.145:0/3701062912 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.196+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f08727f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.195+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087278310 0x55f08727ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.197+0000 7f30b98ad640 1 -- 172.21.15.145:0/3701062912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087278310 msgr2=0x55f08727ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.197+0000 7f30b98ad640 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087278310 0x55f08727ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.197+0000 7f30b98ad640 1 -- 172.21.15.145:0/3701062912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 msgr2=0x55f08727f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.197+0000 7f30b98ad640 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f08727f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.197+0000 7f30b98ad640 1 -- 172.21.15.145:0/3701062912 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f086ef1210 con 0x55f087277920 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.198+0000 7f30b98ad640 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fb0 tx=0x7f30ac00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5c0d8e598c8571 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.198+0000 7f309f7fe640 1 -- 172.21.15.145:0/3701062912 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0357a0 con 0x55f087277920 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.199+0000 7f309f7fe640 1 -- 172.21.15.145:0/3701062912 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ac03b070 con 0x55f087277920 2024-03-28T18:48:49.200 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.199+0000 7f309f7fe640 1 -- 172.21.15.145:0/3701062912 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0342f0 con 0x55f087277920 2024-03-28T18:48:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.200+0000 7f30bdfde740 1 -- 172.21.15.145:0/3701062912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 msgr2=0x55f087277d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.200+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fb0 tx=0x7f30ac00a890 comp rx=0 tx=0).stop 2024-03-28T18:48:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.201+0000 7f30aa7fc640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 0x7f30a00a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.201+0000 7f30aa7fc640 1 -- 172.21.15.145:0/1223200343 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f30b00ced40 con 0x7f30b40756e0 2024-03-28T18:48:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.202+0000 7f30b98ad640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 0x7f30a00a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.212 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30bdfde740 1 -- 172.21.15.145:0/3701062912 shutdown_connections 2024-03-28T18:48:49.212 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30b98ad640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 0x7f30a00a4f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30ac036960 tx=0x7f30ac009c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f08727f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087278310 0x55f08727ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3701062912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087277920 0x55f087277d40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.213 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.212+0000 7f30bdfde740 1 -- 172.21.15.145:0/3701062912 >> 172.21.15.145:0/3701062912 conn(0x55f087272e90 msgr2=0x55f087273290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.217+0000 7f30bdfde740 1 -- 172.21.15.145:0/3701062912 shutdown_connections 2024-03-28T18:48:49.217 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.217+0000 7f30bdfde740 1 -- 172.21.15.145:0/3701062912 wait complete. 2024-03-28T18:48:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.217+0000 7f30bdfde740 1 -- start start 2024-03-28T18:48:49.220 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.217+0000 7f30bdfde740 1 Processor -- start 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 0x55f087282170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f0872891b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f087280430 con 0x55f087278310 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f087198d50 con 0x55f08727d480 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0872805b0 con 0x55f087277920 2024-03-28T18:48:49.221 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30bdfde740 10 client.0 _ll_register_callbacks cb 0x55f087198720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.221+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.222+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57046/0 (socket says 172.21.15.145:57046) 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.222+0000 7f30b90ac640 1 -- 172.21.15.145:0/3906341359 learned_addr learned my addr 172.21.15.145:0/3906341359 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 -- 172.21.15.145:0/3906341359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 msgr2=0x55f0872891b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f0872891b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 -- 172.21.15.145:0/3906341359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 msgr2=0x55f087282170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 0x55f087282170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 0x55f087282170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 -- 172.21.15.145:0/3906341359 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f086fbd620 con 0x55f087277920 2024-03-28T18:48:49.223 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 0x55f087282170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:48:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009900 tx=0x7f30a4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39bc72ac1f2829e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.223+0000 7f309cff9640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:48:49.224 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.224+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4009520 con 0x55f087277920 2024-03-28T18:48:49.225 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.225+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f30a4005000 con 0x55f087277920 2024-03-28T18:48:49.225 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.225+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30ac03b070 con 0x55f087277920 2024-03-28T18:48:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.226+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30a4005200 con 0x55f087277920 2024-03-28T18:48:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.229+0000 7f30bdfde740 10 client.25882 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.229+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f08728c590 con 0x55f087277920 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.232+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ac03b070 con 0x55f087277920 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f30a40035f0 con 0x55f087277920 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 10 client.25882 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 20 client.25882 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f08728dc10 0x55f087290040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 4 client.25882 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:49.234 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.233+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0871ea050 con 0x55f08728dc10 2024-03-28T18:48:49.235 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.234+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f08728dc10 0x55f087290040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.239 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.239+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f08728dc10 0x55f087290040 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d4502b446568bf2 server_cookie=ec27e79c2f118b77 in_seq=0 out_seq=0 2024-03-28T18:48:49.241 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.240+0000 7f309cff9640 10 client.25882 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:49.242 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.241+0000 7f309cff9640 1 -- 172.21.15.145:0/3906341359 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f0871ea050 con 0x55f08728dc10 2024-03-28T18:48:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.241+0000 7f309cff9640 10 client.25882 handle_command_reply: tid=0 2024-03-28T18:48:49.243 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.243+0000 7f30bdfde740 1 client.25882 shutdown 2024-03-28T18:48:49.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.244+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 msgr2=0x55f087281c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.244 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.244+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009900 tx=0x7f30a4004270 comp rx=0 tx=0).stop 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 shutdown_connections 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f08728dc10 0x55f087290040 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f08727d480 0x55f0872891b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f087278310 0x55f087282170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3906341359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087281c30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 >> 172.21.15.145:0/3906341359 conn(0x55f087272e90 msgr2=0x55f087272020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.245 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.245+0000 7f309cff9640 0 client.25882 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:49.250 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.249+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 shutdown_connections 2024-03-28T18:48:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.253+0000 7f30bdfde740 1 -- 172.21.15.145:0/3906341359 wait complete. 2024-03-28T18:48:49.253 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.254+0000 7f30bdfde740 20 client.25882 trim_cache size 0 max 16384 2024-03-28T18:48:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.282+0000 7f30bdfde740 1 Processor -- start 2024-03-28T18:48:49.282 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.282+0000 7f30bdfde740 1 -- start start 2024-03-28T18:48:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.282+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f0872890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087294e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.283 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57048/0 (socket says 172.21.15.145:57048) 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30b90ac640 1 -- 172.21.15.145:0/2370210117 learned_addr learned my addr 172.21.15.145:0/2370210117 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41b0690 con 0x55f08727d480 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41b2010 con 0x55f087278310 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.283+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41b0cf0 con 0x55f087277920 2024-03-28T18:48:49.284 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.284+0000 7f30b98ad640 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087294e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.285+0000 7f30b90ac640 1 -- 172.21.15.145:0/2370210117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 msgr2=0x55f0872890d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:48:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.285+0000 7f30b90ac640 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f0872890d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.285+0000 7f30b90ac640 1 -- 172.21.15.145:0/2370210117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 msgr2=0x55f087294e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.285+0000 7f30b90ac640 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087294e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.285+0000 7f30b90ac640 1 -- 172.21.15.145:0/2370210117 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0872790b0 con 0x55f087277920 2024-03-28T18:48:49.288 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.288+0000 7f30b90ac640 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30a4003960 tx=0x7f30a4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb96a71ff1151092 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.291+0000 7f309effd640 1 -- 172.21.15.145:0/2370210117 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4055020 con 0x55f087277920 2024-03-28T18:48:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.291+0000 7f309effd640 1 -- 172.21.15.145:0/2370210117 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30a4009520 con 0x55f087277920 2024-03-28T18:48:49.291 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.291+0000 7f309effd640 1 -- 172.21.15.145:0/2370210117 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4004a00 con 0x55f087277920 2024-03-28T18:48:49.293 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 msgr2=0x55f087294940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30a4003960 tx=0x7f30a4005e80 comp rx=0 tx=0).stop 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 shutdown_connections 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087294e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f0872890d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 --2- 172.21.15.145:0/2370210117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f087294940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.294 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.293+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 >> 172.21.15.145:0/2370210117 conn(0x55f087272e90 msgr2=0x55f087272020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.296+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 shutdown_connections 2024-03-28T18:48:49.296 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.296+0000 7f30bdfde740 1 -- 172.21.15.145:0/2370210117 wait complete. 2024-03-28T18:48:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.297+0000 7f30bdfde740 1 -- start start 2024-03-28T18:48:49.297 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.297+0000 7f30bdfde740 1 Processor -- start 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f0872888c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f087288e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087286f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f0872888c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30b90ac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f0872888c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:57062/0 (socket says 172.21.15.145:57062) 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30b90ac640 1 -- 172.21.15.145:0/3991949712 learned_addr learned my addr 172.21.15.145:0/3991949712 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41b19b0 con 0x55f08727d480 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41b1350 con 0x55f087278310 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41ae930 con 0x55f087277920 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.298+0000 7f30bdfde740 10 client.0 _ll_register_callbacks cb 0x7f30a402c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:49.299 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.299+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f087288e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 -- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 msgr2=0x55f0872888c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f0872888c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 -- 172.21.15.145:0/3991949712 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 msgr2=0x55f087286f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087286f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.301 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f087287510 con 0x55f087278310 2024-03-28T18:48:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.301+0000 7f30b88ab640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f087288e00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f30b0005fd0 tx=0x7f30b0004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcec9ecdb1ac2087 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:49.302 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.302+0000 7f309cff9640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:48:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.302+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b009ae90 con 0x55f087278310 2024-03-28T18:48:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.302+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f30b0099650 con 0x55f087278310 2024-03-28T18:48:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.302+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30a4009520 con 0x55f087278310 2024-03-28T18:48:49.303 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.304+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b009acb0 con 0x55f087278310 2024-03-28T18:48:49.304 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.305+0000 7f30bdfde740 10 client.25811 fetch_fsmap learned FSMap version 31 2024-03-28T18:48:49.305 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.305+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f0872e9ed0 con 0x55f087278310 2024-03-28T18:48:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30a4009520 con 0x55f087278310 2024-03-28T18:48:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f30b00979c0 con 0x55f087278310 2024-03-28T18:48:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 10 client.25811 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:48:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 10 client.25811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:48:49.306 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 20 client.25811 populate_metadata read hostname 'smithi145' 2024-03-28T18:48:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f0872ea7e0 0x55f0872ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 4 client.25811 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:48:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.306+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f0871e43a0 con 0x55f0872ea7e0 2024-03-28T18:48:49.307 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.307+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f0872ea7e0 0x55f0872ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:49.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.311+0000 7f30b90ac640 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f0872ea7e0 0x55f0872ecc00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b79f27f18591e42 server_cookie=ac927f63d98bbd1d in_seq=0 out_seq=0 2024-03-28T18:48:49.311 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.311+0000 7f309cff9640 10 client.25811 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.339+0000 7f309cff9640 1 -- 172.21.15.145:0/3991949712 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f0871e43a0 con 0x55f0872ea7e0 2024-03-28T18:48:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.339+0000 7f309cff9640 10 client.25811 handle_command_reply: tid=0 2024-03-28T18:48:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.339+0000 7f30bdfde740 1 client.25811 shutdown 2024-03-28T18:48:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.341+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 msgr2=0x55f087288e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.341+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f087288e00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f30b0005fd0 tx=0x7f30b0004570 comp rx=0 tx=0).stop 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 shutdown_connections 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x55f0872ea7e0 0x55f0872ecc00 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f08727d480 0x55f087286f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f087278310 0x55f087288e00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 --2- 172.21.15.145:0/3991949712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f087277920 0x55f0872888c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 >> 172.21.15.145:0/3991949712 conn(0x55f087272e90 msgr2=0x55f0872760f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f309cff9640 0 client.25811 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:48:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 shutdown_connections 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 1 -- 172.21.15.145:0/3991949712 wait complete. 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.342+0000 7f30bdfde740 20 client.25811 trim_cache size 0 max 16384 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:48:49.343 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 msgr2=0x7f30a00a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 0x7f30a00a4f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30ac036960 tx=0x7f30ac009c60 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 msgr2=0x7f30b410a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c890 tx=0x7f30b000cd50 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 shutdown_connections 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30b410dcc0 0x7f30b41ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b40756e0 0x7f30b410a870 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f30a00a2ad0 0x7f30a00a4f90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 --2- 172.21.15.145:0/1223200343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30b4074100 0x7f30b410a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.343+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 >> 172.21.15.145:0/1223200343 conn(0x7f30b40fcc60 msgr2=0x7f30b4101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:49.344 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.344+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 shutdown_connections 2024-03-28T18:48:49.345 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:48:49.344+0000 7f30bb310640 1 -- 172.21.15.145:0/1223200343 wait complete. 2024-03-28T18:48:49.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:49 smithi145 ceph-mon[32151]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 436 MiB/s wr, 3.47k op/s 2024-03-28T18:48:49.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:49 smithi192 ceph-mon[36589]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 436 MiB/s wr, 3.47k op/s 2024-03-28T18:48:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:49 smithi164 ceph-mon[36248]: pgmap v682: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 436 MiB/s wr, 3.47k op/s 2024-03-28T18:48:51.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:51 smithi145 ceph-mon[32151]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 520 MiB/s wr, 3.82k op/s 2024-03-28T18:48:51.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:51 smithi192 ceph-mon[36589]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 520 MiB/s wr, 3.82k op/s 2024-03-28T18:48:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:51 smithi164 ceph-mon[36248]: pgmap v683: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s rd, 520 MiB/s wr, 3.82k op/s 2024-03-28T18:48:53.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:53 smithi145 ceph-mon[32151]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 484 MiB/s wr, 3.25k op/s 2024-03-28T18:48:53.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:53 smithi192 ceph-mon[36589]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 484 MiB/s wr, 3.25k op/s 2024-03-28T18:48:53.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:53 smithi164 ceph-mon[36248]: pgmap v684: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 484 MiB/s wr, 3.25k op/s 2024-03-28T18:48:55.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:55 smithi145 ceph-mon[32151]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 467 MiB/s wr, 2.92k op/s 2024-03-28T18:48:55.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:55 smithi192 ceph-mon[36589]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 467 MiB/s wr, 2.92k op/s 2024-03-28T18:48:55.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:55 smithi164 ceph-mon[36248]: pgmap v685: 129 pgs: 129 active+clean; 10 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 467 MiB/s wr, 2.92k op/s 2024-03-28T18:48:57.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:57 smithi145 ceph-mon[32151]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 558 MiB/s wr, 3.45k op/s 2024-03-28T18:48:57.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:57 smithi192 ceph-mon[36589]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 558 MiB/s wr, 3.45k op/s 2024-03-28T18:48:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:57 smithi164 ceph-mon[36248]: pgmap v686: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s rd, 558 MiB/s wr, 3.45k op/s 2024-03-28T18:48:59.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:48:59 smithi145 ceph-mon[32151]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 454 MiB/s wr, 2.79k op/s 2024-03-28T18:48:59.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:48:59 smithi192 ceph-mon[36589]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 454 MiB/s wr, 2.79k op/s 2024-03-28T18:48:59.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:48:59 smithi164 ceph-mon[36248]: pgmap v687: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 454 MiB/s wr, 2.79k op/s 2024-03-28T18:49:00.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:00 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:00 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:00.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:00 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:01.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:01 smithi192 ceph-mon[36589]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 516 MiB/s wr, 3.27k op/s 2024-03-28T18:49:01.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:01 smithi164 ceph-mon[36248]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 516 MiB/s wr, 3.27k op/s 2024-03-28T18:49:01.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:01 smithi145 ceph-mon[32151]: pgmap v688: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 516 MiB/s wr, 3.27k op/s 2024-03-28T18:49:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcb8b45640 1 -- 172.21.15.145:0/4263780842 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0032340 con 0x7fbcb4109000 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 -- 172.21.15.145:0/4263780842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb4109000 msgr2=0x7fbcb410f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/4263780842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb4109000 0x7fbcb410f9e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009860 tx=0x7fbcb002bb90 comp rx=0 tx=0).stop 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 -- 172.21.15.145:0/4263780842 shutdown_connections 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/4263780842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb410ff20 0x7fbcb4112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/4263780842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb4109000 0x7fbcb410f9e0 secure :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009860 tx=0x7fbcb002bb90 comp rx=0 tx=0).stop 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/4263780842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.554+0000 7fbcbbdab640 1 -- 172.21.15.145:0/4263780842 >> 172.21.15.145:0/4263780842 conn(0x7fbcb406e6e0 msgr2=0x7fbcb4070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.559+0000 7fbcbbdab640 1 -- 172.21.15.145:0/4263780842 shutdown_connections 2024-03-28T18:49:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.561+0000 7fbcbbdab640 1 -- 172.21.15.145:0/4263780842 wait complete. 2024-03-28T18:49:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.562+0000 7fbcbbdab640 1 Processor -- start 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.562+0000 7fbcbbdab640 1 -- start start 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.562+0000 7fbcbbdab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4119540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.562+0000 7fbcbbdab640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 0x7fbcb4119a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.562+0000 7fbcbbdab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcbbdab640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb410b430 con 0x7fbcb410ff20 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcbbdab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb410b2b0 con 0x7fbcb41086a0 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcbbdab640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb410b5b0 con 0x7fbcb411eae0 2024-03-28T18:49:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcba348640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcba348640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:46192/0 (socket says 172.21.15.145:46192) 2024-03-28T18:49:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4119540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcb9346640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 0x7fbcb4119a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.563+0000 7fbcba348640 1 -- 172.21.15.145:0/1859718218 learned_addr learned my addr 172.21.15.145:0/1859718218 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.564+0000 7fbcb9346640 1 -- 172.21.15.145:0/1859718218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 msgr2=0x7fbcb411ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.564+0000 7fbcb9346640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.564+0000 7fbcb9346640 1 -- 172.21.15.145:0/1859718218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 msgr2=0x7fbcb4119540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.564+0000 7fbcb9346640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4119540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.564+0000 7fbcb9346640 1 -- 172.21.15.145:0/1859718218 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb411f550 con 0x7fbcb410ff20 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.565+0000 7fbcba348640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.565+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4119540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.568+0000 7fbcb9346640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 0x7fbcb4119a80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0008000 tx=0x7fbcb0004390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.569+0000 7fbca6ffd640 1 -- 172.21.15.145:0/1859718218 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0039040 con 0x7fbcb410ff20 2024-03-28T18:49:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.569+0000 7fbca6ffd640 1 -- 172.21.15.145:0/1859718218 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcb002ca10 con 0x7fbcb410ff20 2024-03-28T18:49:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.569+0000 7fbca6ffd640 1 -- 172.21.15.145:0/1859718218 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb003d430 con 0x7fbcb410ff20 2024-03-28T18:49:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.570+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb411f7e0 con 0x7fbcb410ff20 2024-03-28T18:49:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.570+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb411fc50 con 0x7fbcb410ff20 2024-03-28T18:49:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.573+0000 7fbcbe98a740 1 Processor -- start 2024-03-28T18:49:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 -- start start 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa79ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7a2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa6ffb10 con 0x5625aa7a0480 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa6a8900 con 0x5625aa79a920 2024-03-28T18:49:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.574+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa6fbbb0 con 0x5625aa79b310 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.575+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.575+0000 7fbcba348640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa79ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.575+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:46204/0 (socket says 172.21.15.145:46204) 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.575+0000 7fbcb9b47640 1 -- 172.21.15.145:0/322115863 learned_addr learned my addr 172.21.15.145:0/322115863 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 -- 172.21.15.145:0/322115863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 msgr2=0x5625aa79ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa79ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 -- 172.21.15.145:0/322115863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 msgr2=0x5625aa7a2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7a2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 -- 172.21.15.145:0/322115863 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625aa414210 con 0x5625aa79b310 2024-03-28T18:49:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbca6ffd640 1 -- 172.21.15.145:0/1859718218 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbcb0044020 con 0x7fbcb410ff20 2024-03-28T18:49:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.576+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00a8e0 tx=0x7fbcac0096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55a0ab79f865a77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.577+0000 7fbca6ffd640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 0x7fbca00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.577+0000 7fbca6ffd640 1 -- 172.21.15.145:0/1859718218 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbcb00f0000 con 0x7fbcb410ff20 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.578+0000 7fbc9ffff640 1 -- 172.21.15.145:0/322115863 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac018020 con 0x5625aa79b310 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.578+0000 7fbc9ffff640 1 -- 172.21.15.145:0/322115863 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcac014400 con 0x5625aa79b310 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.578+0000 7fbc9ffff640 1 -- 172.21.15.145:0/322115863 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac013580 con 0x5625aa79b310 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.579+0000 7fbcba348640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 0x7fbca00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.580+0000 7fbcbe98a740 1 -- 172.21.15.145:0/322115863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 msgr2=0x5625aa79fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.580+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00a8e0 tx=0x7fbcac0096d0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcbe98a740 1 -- 172.21.15.145:0/322115863 shutdown_connections 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7a2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa79fe10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/322115863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa79ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcbe98a740 1 -- 172.21.15.145:0/322115863 >> 172.21.15.145:0/322115863 conn(0x5625aa795e90 msgr2=0x5625aa796290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.581+0000 7fbcba348640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 0x7fbca00a4eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x5625aa79bb70 tx=0x7fbca800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.583+0000 7fbcbe98a740 1 -- 172.21.15.145:0/322115863 shutdown_connections 2024-03-28T18:49:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.583+0000 7fbcbe98a740 1 -- 172.21.15.145:0/322115863 wait complete. 2024-03-28T18:49:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 -- start start 2024-03-28T18:49:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 Processor -- start 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa7a3430 con 0x5625aa7a0480 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa6bbd50 con 0x5625aa79a920 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.584+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625aa7a35b0 con 0x5625aa79b310 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.585+0000 7fbcbe98a740 10 client.0 _ll_register_callbacks cb 0x5625aa6bb720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.585+0000 7fbcb9346640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.585+0000 7fbcb9346640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:46218/0 (socket says 172.21.15.145:46218) 2024-03-28T18:49:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.585+0000 7fbcb9346640 1 -- 172.21.15.145:0/3486175558 learned_addr learned my addr 172.21.15.145:0/3486175558 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcba348640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcb9346640 1 -- 172.21.15.145:0/3486175558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 msgr2=0x5625aa7a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcb9346640 1 -- 172.21.15.145:0/3486175558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 msgr2=0x5625aa7ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.586+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.587+0000 7fbcb9346640 1 -- 172.21.15.145:0/3486175558 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625aa4e0620 con 0x5625aa79b310 2024-03-28T18:49:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.587+0000 7fbcba348640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.587+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00349e0 tx=0x7fbcb0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b68f4b5298f9c5cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.587+0000 7fbc9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:49:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.591+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00bbef0 con 0x5625aa79b310 2024-03-28T18:49:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.591+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbcb00f20f0 con 0x5625aa79b310 2024-03-28T18:49:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.593+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcac014400 con 0x5625aa79b310 2024-03-28T18:49:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.593+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcb00bcc90 con 0x5625aa79b310 2024-03-28T18:49:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.593+0000 7fbcbe98a740 10 client.25900 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.593+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5625aa7a34a0 con 0x5625aa79b310 2024-03-28T18:49:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.594+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcac014400 con 0x5625aa79b310 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbcb003c070 con 0x5625aa79b310 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 10 client.25900 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 10 client.25900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 20 client.25900 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa7b0e50 0x5625aa7b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 4 client.25900 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.595+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625aa70d050 con 0x5625aa7b0e50 2024-03-28T18:49:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.596+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa7b0e50 0x5625aa7b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.603+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa7b0e50 0x5625aa7b3270 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a51e2c22ab512008 server_cookie=ba50e5f7c1132dfe in_seq=0 out_seq=0 2024-03-28T18:49:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.603+0000 7fbc9d7fa640 10 client.25900 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.605+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3486175558 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625aa70d050 con 0x5625aa7b0e50 2024-03-28T18:49:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.605+0000 7fbc9d7fa640 10 client.25900 handle_command_reply: tid=0 2024-03-28T18:49:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.605+0000 7fbcbe98a740 1 client.25900 shutdown 2024-03-28T18:49:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.613+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 msgr2=0x5625aa7a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.613+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00349e0 tx=0x7fbcb0009580 comp rx=0 tx=0).stop 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 shutdown_connections 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa7b0e50 0x5625aa7b3270 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3486175558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 >> 172.21.15.145:0/3486175558 conn(0x5625aa795e90 msgr2=0x5625aa795020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.614+0000 7fbc9d7fa640 0 client.25900 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.616+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 shutdown_connections 2024-03-28T18:49:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.616+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3486175558 wait complete. 2024-03-28T18:49:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.616+0000 7fbcbe98a740 20 client.25900 trim_cache size 0 max 16384 2024-03-28T18:49:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.648+0000 7fbcbe98a740 1 Processor -- start 2024-03-28T18:49:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 -- start start 2024-03-28T18:49:02.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41c1b20 con 0x5625aa7a0480 2024-03-28T18:49:02.651 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41c34a0 con 0x5625aa79b310 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcb9346640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41c2180 con 0x5625aa79a920 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:46232/0 (socket says 172.21.15.145:46232) 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcb9b47640 1 -- 172.21.15.145:0/663675310 learned_addr learned my addr 172.21.15.145:0/663675310 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.649+0000 7fbcba348640 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 -- 172.21.15.145:0/663675310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 msgr2=0x5625aa7a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 -- 172.21.15.145:0/663675310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 msgr2=0x5625aa7ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 -- 172.21.15.145:0/663675310 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625aa796c00 con 0x5625aa79b310 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbcb9346640 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00bc520 tx=0x7fbcb0003a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c81aa3c97986d27e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbc9f7fe640 1 -- 172.21.15.145:0/663675310 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00bbef0 con 0x5625aa79b310 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbc9f7fe640 1 -- 172.21.15.145:0/663675310 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcb00c4d30 con 0x5625aa79b310 2024-03-28T18:49:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.650+0000 7fbc9f7fe640 1 -- 172.21.15.145:0/663675310 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00ede10 con 0x5625aa79b310 2024-03-28T18:49:02.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- 172.21.15.145:0/663675310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 msgr2=0x5625aa7a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00bc520 tx=0x7fbcb0003a00 comp rx=0 tx=0).stop 2024-03-28T18:49:02.656 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- 172.21.15.145:0/663675310 shutdown_connections 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa7a0480 0x5625aa7ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a5250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/663675310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa79a920 0x5625aa7a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- 172.21.15.145:0/663675310 >> 172.21.15.145:0/663675310 conn(0x5625aa795e90 msgr2=0x5625aa7a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- 172.21.15.145:0/663675310 shutdown_connections 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- 172.21.15.145:0/663675310 wait complete. 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.651+0000 7fbcbe98a740 1 -- start start 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 Processor -- start 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa7a0480 0x5625aa7aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41c2e40 con 0x5625aa79a920 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41c27e0 con 0x5625aa79b310 2024-03-28T18:49:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41bfd90 con 0x5625aa7a0480 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcbe98a740 10 client.0 _ll_register_callbacks cb 0x7fbcb00b8170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcb9346640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcb9b47640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54858/0 (socket says 172.21.15.145:54858) 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcb9b47640 1 -- 172.21.15.145:0/3830098979 learned_addr learned my addr 172.21.15.145:0/3830098979 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.652+0000 7fbcba348640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa7a0480 0x5625aa7aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 -- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa7a0480 msgr2=0x5625aa7aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa7a0480 0x5625aa7aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 -- 172.21.15.145:0/3830098979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 msgr2=0x5625aa7a9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 -- 172.21.15.145:0/3830098979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625aa7aaf30 con 0x5625aa79b310 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcb9346640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a9980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5625aa791390 tx=0x7fbcb0009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf55d6e6dbceccc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbc9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb00bbef0 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbcb00bbbf0 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcb00c4d30 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcb00bc360 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcbe98a740 10 client.25826 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.653+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5625aa7aaf30 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcb00c4d30 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fbcb003c070 con 0x5625aa79b310 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 10 client.25826 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 10 client.25826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 20 client.25826 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa80c700 0x5625aa80eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 4 client.25826 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:49:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5625aa7073a0 con 0x5625aa80c700 2024-03-28T18:49:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.654+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa80c700 0x5625aa80eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.655+0000 7fbcb9b47640 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa80c700 0x5625aa80eaf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b6197d33ae9fb8 server_cookie=7d044a57d54a4deb in_seq=0 out_seq=0 2024-03-28T18:49:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.655+0000 7fbc9d7fa640 10 client.25826 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.752+0000 7fbc9d7fa640 1 -- 172.21.15.145:0/3830098979 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5625aa7073a0 con 0x5625aa80c700 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.752+0000 7fbc9d7fa640 10 client.25826 handle_command_reply: tid=0 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.752+0000 7fbcbe98a740 1 client.25826 shutdown 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 msgr2=0x5625aa7a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a9980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5625aa791390 tx=0x7fbcb0009490 comp rx=0 tx=0).stop 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 shutdown_connections 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5625aa80c700 0x5625aa80eaf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5625aa7a0480 0x5625aa7aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625aa79b310 0x5625aa7a9980 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 --2- 172.21.15.145:0/3830098979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625aa79a920 0x5625aa7a9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 >> 172.21.15.145:0/3830098979 conn(0x5625aa795e90 msgr2=0x5625aa794790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.755 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbc9d7fa640 0 client.25826 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 shutdown_connections 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.753+0000 7fbcbe98a740 1 -- 172.21.15.145:0/3830098979 wait complete. 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.754+0000 7fbcbe98a740 20 client.25826 trim_cache size 0 max 16384 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:49:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.755+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 msgr2=0x7fbca00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.755+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 0x7fbca00a4eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x5625aa79bb70 tx=0x7fbca800b390 comp rx=0 tx=0).stop 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.755+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 msgr2=0x7fbcb4119a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.755+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 0x7fbcb4119a80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0008000 tx=0x7fbcb0004390 comp rx=0 tx=0).stop 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 shutdown_connections 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbca00a29f0 0x7fbca00a4eb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbcb411eae0 0x7fbcb411ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcb410ff20 0x7fbcb4119a80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.757 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 --2- 172.21.15.145:0/1859718218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcb41086a0 0x7fbcb4119540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 >> 172.21.15.145:0/1859718218 conn(0x7fbcb406e6e0 msgr2=0x7fbcb40703c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.756+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 shutdown_connections 2024-03-28T18:49:02.758 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:02.757+0000 7fbcbbdab640 1 -- 172.21.15.145:0/1859718218 wait complete. 2024-03-28T18:49:03.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:03 smithi192 ceph-mon[36589]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 458 MiB/s wr, 3.17k op/s 2024-03-28T18:49:03.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:03 smithi164 ceph-mon[36248]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 458 MiB/s wr, 3.17k op/s 2024-03-28T18:49:03.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:03 smithi145 ceph-mon[32151]: pgmap v689: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 458 MiB/s wr, 3.17k op/s 2024-03-28T18:49:05.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:05 smithi192 ceph-mon[36589]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 416 MiB/s wr, 2.98k op/s 2024-03-28T18:49:05.754 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:05 smithi164 ceph-mon[36248]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 416 MiB/s wr, 2.98k op/s 2024-03-28T18:49:05.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:05 smithi145 ceph-mon[32151]: pgmap v690: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 416 MiB/s wr, 2.98k op/s 2024-03-28T18:49:07.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:07 smithi192 ceph-mon[36589]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 467 MiB/s wr, 3.49k op/s 2024-03-28T18:49:07.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:07 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:07.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:07 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:07.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:07 smithi164 ceph-mon[36248]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 467 MiB/s wr, 3.49k op/s 2024-03-28T18:49:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:07 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:07.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:07 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:07.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:07 smithi145 ceph-mon[32151]: pgmap v691: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 467 MiB/s wr, 3.49k op/s 2024-03-28T18:49:07.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:07 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:07.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:07 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:08.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:08 smithi192 ceph-mon[36589]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 360 MiB/s wr, 2.81k op/s 2024-03-28T18:49:08.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:08 smithi164 ceph-mon[36248]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 360 MiB/s wr, 2.81k op/s 2024-03-28T18:49:08.778 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:08 smithi145 ceph-mon[32151]: pgmap v692: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 360 MiB/s wr, 2.81k op/s 2024-03-28T18:49:09.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:09.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:09 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:09.783 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:09.783 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:09 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:09.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:09 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:10.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:10 smithi164 ceph-mon[36248]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 420 MiB/s wr, 3.15k op/s 2024-03-28T18:49:10.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:10 smithi145 ceph-mon[32151]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 420 MiB/s wr, 3.15k op/s 2024-03-28T18:49:10.906 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:10 smithi192 ceph-mon[36589]: pgmap v693: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 420 MiB/s wr, 3.15k op/s 2024-03-28T18:49:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:13 smithi192 ceph-mon[36589]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 404 MiB/s wr, 2.86k op/s 2024-03-28T18:49:13.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:13 smithi164 ceph-mon[36248]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 404 MiB/s wr, 2.86k op/s 2024-03-28T18:49:13.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:13 smithi145 ceph-mon[32151]: pgmap v694: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 404 MiB/s wr, 2.86k op/s 2024-03-28T18:49:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:15 smithi192 ceph-mon[36589]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 367 MiB/s wr, 2.37k op/s 2024-03-28T18:49:15.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:15 smithi164 ceph-mon[36248]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 367 MiB/s wr, 2.37k op/s 2024-03-28T18:49:15.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:15 smithi145 ceph-mon[32151]: pgmap v695: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 367 MiB/s wr, 2.37k op/s 2024-03-28T18:49:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 -- 172.21.15.145:0/4182527264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56381086a0 msgr2=0x7f563810aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 --2- 172.21.15.145:0/4182527264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56381086a0 0x7f563810aad0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f562c01a910 tx=0x7f562c03cca0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 -- 172.21.15.145:0/4182527264 shutdown_connections 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 --2- 172.21.15.145:0/4182527264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810d9c0 0x7f563810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 --2- 172.21.15.145:0/4182527264 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f563810b010 0x7f563810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 --2- 172.21.15.145:0/4182527264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56381086a0 0x7f563810aad0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 -- 172.21.15.145:0/4182527264 >> 172.21.15.145:0/4182527264 conn(0x7f563806e6e0 msgr2=0x7f5638070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 -- 172.21.15.145:0/4182527264 shutdown_connections 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.993+0000 7f563f068640 1 -- 172.21.15.145:0/4182527264 wait complete. 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 Processor -- start 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 -- start start 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 0x7f563819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810b010 0x7f563819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 0x7f56381a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56381119b0 con 0x7f563810b010 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638111830 con 0x7f56381086a0 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.994+0000 7f563f068640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638111b30 con 0x7f563810d9c0 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.996+0000 7f563d605640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 0x7f56381a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.996+0000 7f563d605640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 0x7f56381a5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:42072/0 (socket says 172.21.15.145:42072) 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.996+0000 7f563ce04640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 0x7f563819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.996+0000 7f563d605640 1 -- 172.21.15.145:0/3993005808 learned_addr learned my addr 172.21.15.145:0/3993005808 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f5637fff640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810b010 0x7f563819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f563ce04640 1 -- 172.21.15.145:0/3993005808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 msgr2=0x7f56381a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f563ce04640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 0x7f56381a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f563ce04640 1 -- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810b010 msgr2=0x7f563819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f563ce04640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810b010 0x7f563819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.998+0000 7f563ce04640 1 -- 172.21.15.145:0/3993005808 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56381a5af0 con 0x7f56381086a0 2024-03-28T18:49:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:15.999+0000 7f563ce04640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 0x7f563819ded0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f562c0052a0 tx=0x7f562c045440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.003+0000 7f5635ffb640 1 -- 172.21.15.145:0/3993005808 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c051030 con 0x7f56381086a0 2024-03-28T18:49:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.003+0000 7f5635ffb640 1 -- 172.21.15.145:0/3993005808 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f562c0457c0 con 0x7f56381086a0 2024-03-28T18:49:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.003+0000 7f5635ffb640 1 -- 172.21.15.145:0/3993005808 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c003770 con 0x7f56381086a0 2024-03-28T18:49:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.003+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56381a5d80 con 0x7f56381086a0 2024-03-28T18:49:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.003+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56381a61c0 con 0x7f56381086a0 2024-03-28T18:49:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.007+0000 7f5641c4e740 1 Processor -- start 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 -- start start 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x56217866ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866f310 0x562178673e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621785d3b10 con 0x56217866e920 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56217857c900 con 0x562178674480 2024-03-28T18:49:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5641c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621785cfbb0 con 0x56217866f310 2024-03-28T18:49:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:53742/0 (socket says 172.21.15.145:53742) 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 -- 172.21.15.145:0/537663359 learned_addr learned my addr 172.21.15.145:0/537663359 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 -- 172.21.15.145:0/537663359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866f310 msgr2=0x562178673e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866f310 0x562178673e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 -- 172.21.15.145:0/537663359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 msgr2=0x56217866ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x56217866ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.008+0000 7f5637fff640 1 -- 172.21.15.145:0/537663359 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621782e8210 con 0x562178674480 2024-03-28T18:49:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.009+0000 7f5637fff640 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f563000ae60 tx=0x7f563000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa0eb34eae068e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.015+0000 7f561effd640 1 -- 172.21.15.145:0/537663359 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630011850 con 0x562178674480 2024-03-28T18:49:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.015+0000 7f561effd640 1 -- 172.21.15.145:0/537663359 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563001a480 con 0x562178674480 2024-03-28T18:49:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.016+0000 7f5641c4e740 1 -- 172.21.15.145:0/537663359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 msgr2=0x562178676910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.016+0000 7f5641c4e740 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f563000ae60 tx=0x7f563000f730 comp rx=0 tx=0).stop 2024-03-28T18:49:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.017+0000 7f5635ffb640 1 -- 172.21.15.145:0/3993005808 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f562c054070 con 0x7f56381086a0 2024-03-28T18:49:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.018+0000 7f5635ffb640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 0x7f56200a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.018+0000 7f5635ffb640 1 -- 172.21.15.145:0/3993005808 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f562c04a040 con 0x7f56381086a0 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f5641c4e740 1 -- 172.21.15.145:0/537663359 shutdown_connections 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f5641c4e740 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178674480 0x562178676910 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f5641c4e740 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866f310 0x562178673e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f5641c4e740 1 --2- 172.21.15.145:0/537663359 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x56217866ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f5641c4e740 1 -- 172.21.15.145:0/537663359 >> 172.21.15.145:0/537663359 conn(0x562178669e90 msgr2=0x56217866a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.020+0000 7f563d605640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 0x7f56200a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.021+0000 7f563d605640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 0x7f56200a4f90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x56217866fb70 tx=0x7f5628008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.022+0000 7f5641c4e740 1 -- 172.21.15.145:0/537663359 shutdown_connections 2024-03-28T18:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.024+0000 7f5641c4e740 1 -- 172.21.15.145:0/537663359 wait complete. 2024-03-28T18:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.024+0000 7f5641c4e740 1 -- start start 2024-03-28T18:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.025+0000 7f5641c4e740 1 Processor -- start 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x562178679200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x5621786801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562178677480 con 0x56217866e920 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56217858fd50 con 0x56217866f310 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562178677600 con 0x562178674480 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.026+0000 7f5641c4e740 10 client.0 _ll_register_callbacks cb 0x56217858f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.027+0000 7f563ce04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.027+0000 7f563ce04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:42640/0 (socket says 172.21.15.145:42640) 2024-03-28T18:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.027+0000 7f563ce04640 1 -- 172.21.15.145:0/211415853 learned_addr learned my addr 172.21.15.145:0/211415853 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.028+0000 7f563ce04640 1 -- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 msgr2=0x5621786801d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:49:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.028+0000 7f563ce04640 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x5621786801d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.028+0000 7f563ce04640 1 -- 172.21.15.145:0/211415853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 msgr2=0x562178679200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.028+0000 7f563ce04640 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x562178679200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.028+0000 7f563ce04640 1 -- 172.21.15.145:0/211415853 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621783b4620 con 0x56217866e920 2024-03-28T18:49:16.034 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.033+0000 7f563ce04640 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f562c0d5860 tx=0x7f562c0ccf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e50eb076bcf3c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.035+0000 7f561c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:49:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.035+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0cdcf0 con 0x56217866e920 2024-03-28T18:49:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.035+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f562c005470 con 0x56217866e920 2024-03-28T18:49:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.035+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f563001a480 con 0x56217866e920 2024-03-28T18:49:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.037+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f562c0cd5f0 con 0x56217866e920 2024-03-28T18:49:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.037+0000 7f5641c4e740 10 client.16029 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.037+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562178683540 con 0x56217866e920 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563001a480 con 0x56217866e920 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f562c0d5900 con 0x56217866e920 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 10 client.16029 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 10 client.16029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 20 client.16029 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562178684c80 0x562178687120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 4 client.16029 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.041+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621785e1050 con 0x562178684c80 2024-03-28T18:49:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.042+0000 7f5637fff640 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562178684c80 0x562178687120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.048+0000 7f5637fff640 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562178684c80 0x562178687120 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbb5c325856a0b32 server_cookie=fbff16e4607a740 in_seq=0 out_seq=0 2024-03-28T18:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.048+0000 7f561c7f8640 10 client.16029 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.052+0000 7f561c7f8640 1 -- 172.21.15.145:0/211415853 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5621785e1050 con 0x562178684c80 2024-03-28T18:49:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.052+0000 7f561c7f8640 10 client.16029 handle_command_reply: tid=0 2024-03-28T18:49:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.052+0000 7f5641c4e740 1 client.16029 shutdown 2024-03-28T18:49:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.052+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 msgr2=0x562178678cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.052+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f562c0d5860 tx=0x7f562c0ccf50 comp rx=0 tx=0).stop 2024-03-28T18:49:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 shutdown_connections 2024-03-28T18:49:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x562178684c80 0x562178687120 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x5621786801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x562178679200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 --2- 172.21.15.145:0/211415853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217866e920 0x562178678cc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 >> 172.21.15.145:0/211415853 conn(0x562178669e90 msgr2=0x562178669020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:16.070 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.069+0000 7f561c7f8640 0 client.16029 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.075+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 shutdown_connections 2024-03-28T18:49:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.075+0000 7f5641c4e740 1 -- 172.21.15.145:0/211415853 wait complete. 2024-03-28T18:49:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.075+0000 7f5641c4e740 20 client.16029 trim_cache size 0 max 16384 2024-03-28T18:49:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 Processor -- start 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 -- start start 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562178669e90 0x56217866d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866e920 0x562178680190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400fae0 con 0x562178669e90 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5624011460 con 0x56217866f310 2024-03-28T18:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.126+0000 7f5641c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5624010140 con 0x56217866e920 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.128+0000 7f563d605640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.128+0000 7f563d605640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:53758/0 (socket says 172.21.15.145:53758) 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.128+0000 7f563d605640 1 -- 172.21.15.145:0/358950961 learned_addr learned my addr 172.21.15.145:0/358950961 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 -- 172.21.15.145:0/358950961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866e920 msgr2=0x562178680190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866e920 0x562178680190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 -- 172.21.15.145:0/358950961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562178669e90 msgr2=0x56217866d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562178669e90 0x56217866d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 -- 172.21.15.145:0/358950961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562178681650 con 0x56217866f310 2024-03-28T18:49:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.129+0000 7f563d605640 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f562800d8e0 tx=0x7f5628010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e90a555aa6adbcd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.131+0000 7f561e7fc640 1 -- 172.21.15.145:0/358950961 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628024070 con 0x56217866f310 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.131+0000 7f561e7fc640 1 -- 172.21.15.145:0/358950961 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5628014b40 con 0x56217866f310 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 -- 172.21.15.145:0/358950961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 msgr2=0x56217866d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f562800d8e0 tx=0x7f5628010000 comp rx=0 tx=0).stop 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 -- 172.21.15.145:0/358950961 shutdown_connections 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56217866f310 0x56217866d630 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56217866e920 0x562178680190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 --2- 172.21.15.145:0/358950961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562178669e90 0x56217866d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.132+0000 7f5641c4e740 1 -- 172.21.15.145:0/358950961 >> 172.21.15.145:0/358950961 conn(0x562178673f50 msgr2=0x562178664eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:16.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.136+0000 7f5641c4e740 1 -- 172.21.15.145:0/358950961 shutdown_connections 2024-03-28T18:49:16.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.136+0000 7f5641c4e740 1 -- 172.21.15.145:0/358950961 wait complete. 2024-03-28T18:49:16.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.137+0000 7f5641c4e740 1 -- start start 2024-03-28T18:49:16.137 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.137+0000 7f5641c4e740 1 Processor -- start 2024-03-28T18:49:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.137+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x56217867e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217867f040 0x56217867f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5624010e00 con 0x56217867f040 2024-03-28T18:49:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240107a0 con 0x562178669e90 2024-03-28T18:49:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400dca0 con 0x562178674480 2024-03-28T18:49:16.139 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.138+0000 7f5641c4e740 10 client.0 _ll_register_callbacks cb 0x7f562c0cd490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.140+0000 7f5637fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x56217867e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.140+0000 7f563ce04640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.140+0000 7f563ce04640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:53766/0 (socket says 172.21.15.145:53766) 2024-03-28T18:49:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.140+0000 7f563ce04640 1 -- 172.21.15.145:0/4209384267 learned_addr learned my addr 172.21.15.145:0/4209384267 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:16.141 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.141+0000 7f563d605640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217867f040 0x56217867f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f5637fff640 1 -- 172.21.15.145:0/4209384267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 msgr2=0x56217867daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f5637fff640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f5637fff640 1 -- 172.21.15.145:0/4209384267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217867f040 msgr2=0x56217867f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f5637fff640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217867f040 0x56217867f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f5637fff640 1 -- 172.21.15.145:0/4209384267 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56217867fa00 con 0x562178674480 2024-03-28T18:49:16.143 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.143+0000 7f563ce04640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.152+0000 7f5637fff640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x56217867e030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f563000fc30 tx=0x7f563000a990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974d30eebc5cd8d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.153+0000 7f561c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-28T18:49:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.153+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563000aa80 con 0x562178674480 2024-03-28T18:49:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.153+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5630004bf0 con 0x562178674480 2024-03-28T18:49:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.156+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621785db3a0 con 0x562178674480 2024-03-28T18:49:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.160+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56300252d0 con 0x562178674480 2024-03-28T18:49:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.160+0000 7f5641c4e740 10 client.25915 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.160+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5621786e0cb0 con 0x562178674480 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5621785db3a0 con 0x562178674480 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5630025b70 con 0x562178674480 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 10 client.25915 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 10 client.25915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 20 client.25915 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5621786e1600 0x5621786e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 4 client.25915 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:49:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.164+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562178559710 con 0x5621786e1600 2024-03-28T18:49:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.173+0000 7f563ce04640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5621786e1600 0x5621786e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.186+0000 7f563ce04640 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5621786e1600 0x5621786e3a20 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e3373ccef23bdaa server_cookie=5229c9d7a25ef9ab in_seq=0 out_seq=0 2024-03-28T18:49:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:16.187+0000 7f561c7f8640 10 client.25915 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.174+0000 7f561c7f8640 1 -- 172.21.15.145:0/4209384267 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562178559710 con 0x5621786e1600 2024-03-28T18:49:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.174+0000 7f561c7f8640 10 client.25915 handle_command_reply: tid=0 2024-03-28T18:49:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.174+0000 7f5641c4e740 1 client.25915 shutdown 2024-03-28T18:49:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.174+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 msgr2=0x56217867e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.174+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x56217867e030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f563000fc30 tx=0x7f563000a990 comp rx=0 tx=0).stop 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 shutdown_connections 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x5621786e1600 0x5621786e3a20 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56217867f040 0x56217867f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562178674480 0x56217867e030 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 --2- 172.21.15.145:0/4209384267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562178669e90 0x56217867daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 >> 172.21.15.145:0/4209384267 conn(0x562178673f50 msgr2=0x562178670560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:17.176 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.176+0000 7f561c7f8640 0 client.25915 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.178+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 shutdown_connections 2024-03-28T18:49:17.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.179+0000 7f5641c4e740 1 -- 172.21.15.145:0/4209384267 wait complete. 2024-03-28T18:49:17.179 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.179+0000 7f5641c4e740 20 client.25915 trim_cache size 0 max 16384 2024-03-28T18:49:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:49:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:49:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:49:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:49:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.182+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 msgr2=0x7f56200a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.182+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 0x7f56200a4f90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x56217866fb70 tx=0x7f5628008040 comp rx=0 tx=0).stop 2024-03-28T18:49:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.182+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 msgr2=0x7f563819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.182+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 0x7f563819ded0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f562c0052a0 tx=0x7f562c045440 comp rx=0 tx=0).stop 2024-03-28T18:49:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 shutdown_connections 2024-03-28T18:49:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f56200a2ad0 0x7f56200a4f90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f563810d9c0 0x7f56381a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563810b010 0x7f563819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 --2- 172.21.15.145:0/3993005808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56381086a0 0x7f563819ded0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.186+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 >> 172.21.15.145:0/3993005808 conn(0x7f563806e6e0 msgr2=0x7f5638108ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:17.187 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.187+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 shutdown_connections 2024-03-28T18:49:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:17.190+0000 7f563f068640 1 -- 172.21.15.145:0/3993005808 wait complete. 2024-03-28T18:49:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:17 smithi192 ceph-mon[36589]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 440 MiB/s wr, 2.74k op/s 2024-03-28T18:49:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:17 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:17 smithi164 ceph-mon[36248]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 440 MiB/s wr, 2.74k op/s 2024-03-28T18:49:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:17 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:17.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:17 smithi145 ceph-mon[32151]: pgmap v696: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 440 MiB/s wr, 2.74k op/s 2024-03-28T18:49:17.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:17.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:17 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:19 smithi192 ceph-mon[36589]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 391 MiB/s wr, 2.33k op/s 2024-03-28T18:49:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:19 smithi164 ceph-mon[36248]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 391 MiB/s wr, 2.33k op/s 2024-03-28T18:49:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:19 smithi145 ceph-mon[32151]: pgmap v697: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 391 MiB/s wr, 2.33k op/s 2024-03-28T18:49:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:49:21.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:21 smithi192 ceph-mon[36589]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 422 MiB/s wr, 2.39k op/s 2024-03-28T18:49:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:21 smithi164 ceph-mon[36248]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 422 MiB/s wr, 2.39k op/s 2024-03-28T18:49:21.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:21 smithi145 ceph-mon[32151]: pgmap v698: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 422 MiB/s wr, 2.39k op/s 2024-03-28T18:49:23.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:23 smithi192 ceph-mon[36589]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 395 MiB/s wr, 1.96k op/s 2024-03-28T18:49:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:23 smithi164 ceph-mon[36248]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 395 MiB/s wr, 1.96k op/s 2024-03-28T18:49:23.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:23 smithi145 ceph-mon[32151]: pgmap v699: 129 pgs: 129 active+clean; 10 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 395 MiB/s wr, 1.96k op/s 2024-03-28T18:49:25.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:25 smithi192 ceph-mon[36589]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 378 MiB/s wr, 1.58k op/s 2024-03-28T18:49:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:25 smithi164 ceph-mon[36248]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 378 MiB/s wr, 1.58k op/s 2024-03-28T18:49:25.531 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:25 smithi145 ceph-mon[32151]: pgmap v700: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 378 MiB/s wr, 1.58k op/s 2024-03-28T18:49:27.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:27 smithi192 ceph-mon[36589]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 468 MiB/s wr, 1.53k op/s 2024-03-28T18:49:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:27 smithi164 ceph-mon[36248]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 468 MiB/s wr, 1.53k op/s 2024-03-28T18:49:27.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:27 smithi145 ceph-mon[32151]: pgmap v701: 129 pgs: 129 active+clean; 10 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 468 MiB/s wr, 1.53k op/s 2024-03-28T18:49:29.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:29 smithi192 ceph-mon[36589]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 451 MiB/s wr, 1.02k op/s 2024-03-28T18:49:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:29 smithi164 ceph-mon[36248]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 451 MiB/s wr, 1.02k op/s 2024-03-28T18:49:29.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:29 smithi145 ceph-mon[32151]: pgmap v702: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 451 MiB/s wr, 1.02k op/s 2024-03-28T18:49:30.491 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.491+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/784488978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e08109390 msgr2=0x7f6e0810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.491+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/784488978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e08109390 0x7f6e0810e000 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009900 tx=0x7f6e0402bc50 comp rx=0 tx=0).stop 2024-03-28T18:49:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.492+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/784488978 shutdown_connections 2024-03-28T18:49:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.492+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/784488978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e08109390 0x7f6e0810e000 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.492 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.492+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/784488978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e080732b0 0x7f6e08108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.492+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/784488978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e08072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.492+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/784488978 >> 172.21.15.145:0/784488978 conn(0x7f6e0806e510 msgr2=0x7f6e08070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.495+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/784488978 shutdown_connections 2024-03-28T18:49:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.499+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/784488978 wait complete. 2024-03-28T18:49:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.500+0000 7f6e0ecf9640 1 Processor -- start 2024-03-28T18:49:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 -- start start 2024-03-28T18:49:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e081181c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 0x7f6e08118700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e08109390 0x7f6e081195b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081107e0 con 0x7f6e08072950 2024-03-28T18:49:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08110660 con 0x7f6e080732b0 2024-03-28T18:49:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.505+0000 7f6e0ecf9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08110960 con 0x7f6e08109390 2024-03-28T18:49:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.508+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e081181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.509+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e081181c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40986/0 (socket says 172.21.15.145:40986) 2024-03-28T18:49:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.509+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/434040222 learned_addr learned my addr 172.21.15.145:0/434040222 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.509+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e08109390 0x7f6e081195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.509+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 0x7f6e08118700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.513+0000 7f6e0d4f6640 1 -- 172.21.15.145:0/434040222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e08109390 msgr2=0x7f6e081195b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.513+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e08109390 0x7f6e081195b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.513+0000 7f6e0d4f6640 1 -- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 msgr2=0x7f6e081181c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.513+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e081181c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.513+0000 7f6e0d4f6640 1 -- 172.21.15.145:0/434040222 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08119d30 con 0x7f6e080732b0 2024-03-28T18:49:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.515+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 0x7f6e08118700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000ccb0 tx=0x7f6e000075f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.516+0000 7f6dfeffd640 1 -- 172.21.15.145:0/434040222 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00004b20 con 0x7f6e080732b0 2024-03-28T18:49:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.516+0000 7f6dfeffd640 1 -- 172.21.15.145:0/434040222 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e00004cc0 con 0x7f6e080732b0 2024-03-28T18:49:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.517+0000 7f6dfeffd640 1 -- 172.21.15.145:0/434040222 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000f690 con 0x7f6e080732b0 2024-03-28T18:49:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.517+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0811a020 con 0x7f6e080732b0 2024-03-28T18:49:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.517+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e081b72c0 con 0x7f6e080732b0 2024-03-28T18:49:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.521+0000 7f6e12c25740 1 Processor -- start 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 -- start start 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd83b20 0x558d1dd83f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd8bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dce8b10 con 0x558d1dd89680 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dc91900 con 0x558d1dd84510 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.524+0000 7f6e12c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dce4bb0 con 0x558d1dd83b20 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52310/0 (socket says 172.21.15.145:52310) 2024-03-28T18:49:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/43737285 learned_addr learned my addr 172.21.15.145:0/43737285 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/43737285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd83b20 msgr2=0x558d1dd83f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd83b20 0x558d1dd83f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/43737285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 msgr2=0x558d1dd8bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd8bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/43737285 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d1d9fd210 con 0x558d1dd84510 2024-03-28T18:49:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.525+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009d70 tx=0x7f6df8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145be24fe75eb1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.527+0000 7f6dfeffd640 1 -- 172.21.15.145:0/434040222 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6e0000f8f0 con 0x7f6e080732b0 2024-03-28T18:49:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.528+0000 7f6dfeffd640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 0x7f6df40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.529+0000 7f6dfeffd640 1 -- 172.21.15.145:0/434040222 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6e000d0f80 con 0x7f6e080732b0 2024-03-28T18:49:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.529+0000 7f6df3fff640 1 -- 172.21.15.145:0/43737285 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8005220 con 0x558d1dd84510 2024-03-28T18:49:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.529+0000 7f6df3fff640 1 -- 172.21.15.145:0/43737285 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6df8004c80 con 0x558d1dd84510 2024-03-28T18:49:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.529+0000 7f6df3fff640 1 -- 172.21.15.145:0/43737285 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8019a20 con 0x558d1dd84510 2024-03-28T18:49:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.531+0000 7f6e12c25740 1 -- 172.21.15.145:0/43737285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 msgr2=0x558d1dd89010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.531+0000 7f6e12c25740 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009d70 tx=0x7f6df8011810 comp rx=0 tx=0).stop 2024-03-28T18:49:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.532+0000 7f6e12c25740 1 -- 172.21.15.145:0/43737285 shutdown_connections 2024-03-28T18:49:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.532+0000 7f6e12c25740 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd8bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.532+0000 7f6e12c25740 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd84510 0x558d1dd89010 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.532+0000 7f6e12c25740 1 --2- 172.21.15.145:0/43737285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd83b20 0x558d1dd83f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.532+0000 7f6e12c25740 1 -- 172.21.15.145:0/43737285 >> 172.21.15.145:0/43737285 conn(0x558d1dd7f090 msgr2=0x558d1dd7f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.535+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 0x7f6df40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.539+0000 7f6e12c25740 1 -- 172.21.15.145:0/43737285 shutdown_connections 2024-03-28T18:49:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.539+0000 7f6e12c25740 1 -- 172.21.15.145:0/43737285 wait complete. 2024-03-28T18:49:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.539+0000 7f6e12c25740 1 -- start start 2024-03-28T18:49:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.539+0000 7f6e12c25740 1 Processor -- start 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd8e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd95380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dd8c630 con 0x558d1dd89680 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dca4d50 con 0x558d1dd83b20 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d1dd8c7b0 con 0x558d1dd84510 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e12c25740 10 client.0 _ll_register_callbacks cb 0x558d1dca4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52314/0 (socket says 172.21.15.145:52314) 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/1125094807 learned_addr learned my addr 172.21.15.145:0/1125094807 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.540+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd8e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 msgr2=0x558d1dd8e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd8e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/1125094807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 msgr2=0x558d1dd95380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd95380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/1125094807 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d1dac9620 con 0x558d1dd83b20 2024-03-28T18:49:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.541+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6df8000c00 tx=0x7f6df8009a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8945d701eb3884ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.548+0000 7f6df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-28T18:49:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.548+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8029020 con 0x558d1dd83b20 2024-03-28T18:49:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.548+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6df8004c80 con 0x558d1dd83b20 2024-03-28T18:49:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.548+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mon.2 v2:172.21.15.164:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6df8004450 con 0x558d1dd83b20 2024-03-28T18:49:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.548+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6df8034050 con 0x558d1dd83b20 2024-03-28T18:49:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.551+0000 7f6e12c25740 10 client.25850 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:30.551 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.551+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558d1dd986f0 con 0x558d1dd83b20 2024-03-28T18:49:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mon.2 v2:172.21.15.164:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6df8004c80 con 0x558d1dd83b20 2024-03-28T18:49:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mon.2 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f6df80035f0 con 0x558d1dd83b20 2024-03-28T18:49:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 0x7f6df40a4ec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006010 tx=0x7f6e04004920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 10 client.25850 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 10 client.25850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 20 client.25850 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1dd9a1a0 0x558d1dd9c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 4 client.25850 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.554+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d1dcf6050 con 0x558d1dd9a1a0 2024-03-28T18:49:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.559+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1dd9a1a0 0x558d1dd9c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.574+0000 7f6df17fa640 10 client.25850 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.574+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1dd9a1a0 0x558d1dd9c5c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c4fc84a02c2b41 server_cookie=d199f96c6128065a in_seq=0 out_seq=0 2024-03-28T18:49:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.578+0000 7f6df17fa640 1 -- 172.21.15.145:0/1125094807 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558d1dcf6050 con 0x558d1dd9a1a0 2024-03-28T18:49:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.578+0000 7f6df17fa640 10 client.25850 handle_command_reply: tid=0 2024-03-28T18:49:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.578+0000 7f6e12c25740 1 client.25850 shutdown 2024-03-28T18:49:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.587+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 msgr2=0x558d1dd8de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.587+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6df8000c00 tx=0x7f6df8009a90 comp rx=0 tx=0).stop 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.588+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 shutdown_connections 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.588+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1dd9a1a0 0x558d1dd9c5c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.589+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd89680 0x558d1dd95380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.589+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd8e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.589+0000 7f6e12c25740 1 --2- 172.21.15.145:0/1125094807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd8de00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.589+0000 7f6df17fa640 0 client.25850 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.589+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 >> 172.21.15.145:0/1125094807 conn(0x558d1dd7f090 msgr2=0x558d1dd7e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.590+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 shutdown_connections 2024-03-28T18:49:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.591+0000 7f6e12c25740 1 -- 172.21.15.145:0/1125094807 wait complete. 2024-03-28T18:49:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.596+0000 7f6e12c25740 20 client.25850 trim_cache size 0 max 16384 2024-03-28T18:49:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.658+0000 7f6e12c25740 1 Processor -- start 2024-03-28T18:49:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.658+0000 7f6e12c25740 1 -- start start 2024-03-28T18:49:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd88dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 0x558d1dd7da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 0x558d1dd7df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec00fae0 con 0x558d1dd84510 2024-03-28T18:49:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec011460 con 0x558d1dd83b20 2024-03-28T18:49:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e12c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec010140 con 0x558d1dd89680 2024-03-28T18:49:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e0d4f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 0x558d1dd7da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e0d4f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 0x558d1dd7da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41018/0 (socket says 172.21.15.145:41018) 2024-03-28T18:49:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.659+0000 7f6e0d4f6640 1 -- 172.21.15.145:0/784190409 learned_addr learned my addr 172.21.15.145:0/784190409 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 0x558d1dd7df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 -- 172.21.15.145:0/784190409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 msgr2=0x558d1dd88dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd88dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 -- 172.21.15.145:0/784190409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 msgr2=0x558d1dd7da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 0x558d1dd7da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.660+0000 7f6e0e4f8640 1 -- 172.21.15.145:0/784190409 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d1dd8ef80 con 0x558d1dd89680 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.661+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 0x558d1dd7df40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e04040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92bd0d03a850ef92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.661+0000 7f6df37fe640 1 -- 172.21.15.145:0/784190409 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e04004f10 con 0x558d1dd89680 2024-03-28T18:49:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.661+0000 7f6df37fe640 1 -- 172.21.15.145:0/784190409 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e04052070 con 0x558d1dd89680 2024-03-28T18:49:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.661+0000 7f6df37fe640 1 -- 172.21.15.145:0/784190409 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0404dc70 con 0x558d1dd89680 2024-03-28T18:49:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.663+0000 7f6e12c25740 1 -- 172.21.15.145:0/784190409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 msgr2=0x558d1dd7df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.663+0000 7f6e12c25740 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 0x558d1dd7df40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009850 tx=0x7f6e04040000 comp rx=0 tx=0).stop 2024-03-28T18:49:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.665+0000 7f6e12c25740 1 -- 172.21.15.145:0/784190409 shutdown_connections 2024-03-28T18:49:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.665+0000 7f6e12c25740 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd89680 0x558d1dd7df40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.665+0000 7f6e12c25740 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd84510 0x558d1dd7da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.665+0000 7f6e12c25740 1 --2- 172.21.15.145:0/784190409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd83b20 0x558d1dd88dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.665+0000 7f6e12c25740 1 -- 172.21.15.145:0/784190409 >> 172.21.15.145:0/784190409 conn(0x558d1dd7f090 msgr2=0x558d1dd7f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.672+0000 7f6e12c25740 1 -- 172.21.15.145:0/784190409 shutdown_connections 2024-03-28T18:49:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.672+0000 7f6e12c25740 1 -- 172.21.15.145:0/784190409 wait complete. 2024-03-28T18:49:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.673+0000 7f6e12c25740 1 -- start start 2024-03-28T18:49:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.673+0000 7f6e12c25740 1 Processor -- start 2024-03-28T18:49:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.673+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.673+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd92110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e12c25740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd89680 0x558d1dd93120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e12c25740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec010e00 con 0x558d1dd83b20 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e12c25740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec0107a0 con 0x558d1dd89680 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e12c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec00dc80 con 0x558d1dd84510 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e12c25740 10 client.0 _ll_register_callbacks cb 0x7f6df80049d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e0dcf7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41024/0 (socket says 172.21.15.145:41024) 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/2980458020 learned_addr learned my addr 172.21.15.145:0/2980458020 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:30.675 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd92110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.674+0000 7f6e0e4f8640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd89680 0x558d1dd93120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.675+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 msgr2=0x558d1dd92110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.675+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd92110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.675+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/2980458020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd89680 msgr2=0x558d1dd93120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.675+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd89680 0x558d1dd93120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.675+0000 7f6e0dcf7640 1 -- 172.21.15.145:0/2980458020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d1dd93710 con 0x558d1dd83b20 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.676+0000 7f6e0dcf7640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6df8011850 tx=0x7f6df8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f6970fc898a90f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.676+0000 7f6df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.676+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8029040 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.676+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e04052070 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.676+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mon.0 v2:172.21.15.145:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6df8025470 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.677+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6df8025670 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.677+0000 7f6e12c25740 10 client.16044 fetch_fsmap learned FSMap version 31 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.677+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558d1dd94a20 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mon.0 v2:172.21.15.145:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e04052070 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mon.0 v2:172.21.15.145:3300/0 5 ==== fsmap(e 31) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f6df8019a10 con 0x558d1dd83b20 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 10 client.16044 fetch_fsmap finished waiting for FSMap version 31 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T18:49:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 20 client.16044 populate_metadata read hostname 'smithi145' 2024-03-28T18:49:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1ddf5c80 0x558d1ddf80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 4 client.16044 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T18:49:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 --> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558d1dcf03a0 con 0x558d1ddf5c80 2024-03-28T18:49:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.678+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1ddf5c80 0x558d1ddf80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.680+0000 7f6e0d4f6640 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1ddf5c80 0x558d1ddf80a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425947c2dd889060 server_cookie=dd9f7371a82e28c6 in_seq=0 out_seq=0 2024-03-28T18:49:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:30.682+0000 7f6df17fa640 10 client.16044 ms_handle_connect on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:31 smithi164 ceph-mon[36248]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 MiB/s wr, 847 op/s 2024-03-28T18:49:31.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:31 smithi145 ceph-mon[32151]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 MiB/s wr, 847 op/s 2024-03-28T18:49:31.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:31 smithi192 ceph-mon[36589]: pgmap v703: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 557 MiB/s wr, 847 op/s 2024-03-28T18:49:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:33 smithi164 ceph-mon[36248]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 548 MiB/s wr, 534 op/s 2024-03-28T18:49:33.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:33 smithi145 ceph-mon[32151]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 548 MiB/s wr, 534 op/s 2024-03-28T18:49:33.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:33 smithi192 ceph-mon[36589]: pgmap v704: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 548 MiB/s wr, 534 op/s 2024-03-28T18:49:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:35 smithi164 ceph-mon[36248]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 555 MiB/s wr, 520 op/s 2024-03-28T18:49:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:35 smithi145 ceph-mon[32151]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 555 MiB/s wr, 520 op/s 2024-03-28T18:49:35.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:35 smithi192 ceph-mon[36589]: pgmap v705: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 555 MiB/s wr, 520 op/s 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.457+0000 7f6df17fa640 1 -- 172.21.15.145:0/2980458020 <== mds.0 v2:172.21.15.192:6838/2516430938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558d1dcf03a0 con 0x558d1ddf5c80 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.457+0000 7f6df17fa640 10 client.16044 handle_command_reply: tid=0 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.457+0000 7f6e12c25740 1 client.16044 shutdown 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 msgr2=0x558d1dd91bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6df8011850 tx=0x7f6df8003820 comp rx=0 tx=0).stop 2024-03-28T18:49:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 shutdown_connections 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:6838/2516430938,v1:172.21.15.192:6839/2516430938] conn(0x558d1ddf5c80 0x558d1ddf80a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558d1dd89680 0x558d1dd93120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x558d1dd84510 0x558d1dd92110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 --2- 172.21.15.145:0/2980458020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558d1dd83b20 0x558d1dd91bd0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 >> 172.21.15.145:0/2980458020 conn(0x558d1dd7f090 msgr2=0x558d1dd88c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.458+0000 7f6df17fa640 0 client.16044 ms_handle_reset on v2:172.21.15.192:6838/2516430938 2024-03-28T18:49:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.475+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 shutdown_connections 2024-03-28T18:49:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.475+0000 7f6e12c25740 1 -- 172.21.15.145:0/2980458020 wait complete. 2024-03-28T18:49:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.476+0000 7f6e12c25740 20 client.16044 trim_cache size 0 max 16384 2024-03-28T18:49:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:{ 2024-03-28T18:49:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "message": "", 2024-03-28T18:49:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout: "return_code": 0 2024-03-28T18:49:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stdout:} 2024-03-28T18:49:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.477+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 msgr2=0x7f6df40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.477+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 0x7f6df40a4ec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006010 tx=0x7f6e04004920 comp rx=0 tx=0).stop 2024-03-28T18:49:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.477+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 msgr2=0x7f6e08118700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.477+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 0x7f6e08118700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000ccb0 tx=0x7f6e000075f0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.479+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 shutdown_connections 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.479+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6df40a2a00 0x7f6df40a4ec0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.480+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6e08109390 0x7f6e081195b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.480+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e080732b0 0x7f6e08118700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.480+0000 7f6e0ecf9640 1 --2- 172.21.15.145:0/434040222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e08072950 0x7f6e081181c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.480+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 >> 172.21.15.145:0/434040222 conn(0x7f6e0806e510 msgr2=0x7f6e0810c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.486+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 shutdown_connections 2024-03-28T18:49:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi145.front.sepia.ceph.com.smithi145.stderr:2024-03-28T18:49:36.486+0000 7f6e0ecf9640 1 -- 172.21.15.145:0/434040222 wait complete. 2024-03-28T18:49:37.531 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:37 smithi145 ceph-mon[32151]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 620 MiB/s wr, 741 op/s 2024-03-28T18:49:37.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:37 smithi192 ceph-mon[36589]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 620 MiB/s wr, 741 op/s 2024-03-28T18:49:37.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:37 smithi164 ceph-mon[36248]: pgmap v706: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 620 MiB/s wr, 741 op/s 2024-03-28T18:49:38.391 INFO:tasks.workunit.client.0.smithi145.stdout:stopping iogen 2024-03-28T18:49:38.391 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo 'stopping iogen' 2024-03-28T18:49:38.391 INFO:tasks.workunit.client.0.smithi145.stderr:+ ./iogen -k 2024-03-28T18:49:38.398 INFO:tasks.workunit.client.0.smithi145.stdout:OK 2024-03-28T18:49:38.399 INFO:tasks.workunit.client.0.smithi145.stderr:+ echo OK 2024-03-28T18:49:39.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:39 smithi192 ceph-mon[36589]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 547 MiB/s wr, 728 op/s 2024-03-28T18:49:39.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:39 smithi164 ceph-mon[36248]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 547 MiB/s wr, 728 op/s 2024-03-28T18:49:39.780 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:39 smithi145 ceph-mon[32151]: pgmap v707: 129 pgs: 129 active+clean; 10 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 547 MiB/s wr, 728 op/s 2024-03-28T18:49:40.618 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:49:40.619 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T18:49:41.039 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-03-28T18:49:41.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T18:49:41.385 DEBUG:teuthology.parallel:result is None 2024-03-28T18:49:41.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T18:49:41.420 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T18:49:41.420 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T18:49:41.434 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:49:41.470 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T18:49:41.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:49:41.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 -- 172.21.15.145:0/2735380565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 msgr2=0x7f3254100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 --2- 172.21.15.145:0/2735380565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254100a00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3244009900 tx=0x7f324402bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 -- 172.21.15.145:0/2735380565 shutdown_connections 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 --2- 172.21.15.145:0/2735380565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f3254108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 --2- 172.21.15.145:0/2735380565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 0x7f3254101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 --2- 172.21.15.145:0/2735380565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254100a00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 -- 172.21.15.145:0/2735380565 >> 172.21.15.145:0/2735380565 conn(0x7f32540fbb50 msgr2=0x7f32540fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:41.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 -- 172.21.15.145:0/2735380565 shutdown_connections 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.637+0000 7f3258922640 1 -- 172.21.15.145:0/2735380565 wait complete. 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.638+0000 7f3258922640 1 Processor -- start 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.638+0000 7f3258922640 1 -- start start 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 0x7f325419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325410c250 con 0x7f32541017d0 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325410c0d0 con 0x7f32541005e0 2024-03-28T18:49:41.642 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3258922640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325410c3d0 con 0x7f3254102190 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3252d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3249d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 0x7f325419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:37214/0 (socket says 172.21.15.145:37214) 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3252d9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:35534/0 (socket says 172.21.15.145:35534) 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 -- 172.21.15.145:0/3884319453 learned_addr learned my addr 172.21.15.145:0/3884319453 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 -- 172.21.15.145:0/3884319453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 msgr2=0x7f32541a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.643 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 -- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 msgr2=0x7f325419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 0x7f325419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 -- 172.21.15.145:0/3884319453 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32541a1880 con 0x7f32541005e0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f3252d9d640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f325259c640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f3244004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244041070 con 0x7f32541005e0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.639+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3244004450 con 0x7f32541005e0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.640+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244003df0 con 0x7f32541005e0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.640+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32541a1b10 con 0x7f32541005e0 2024-03-28T18:49:41.644 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.640+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32541a2000 con 0x7f32541005e0 2024-03-28T18:49:41.645 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.642+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3244044070 con 0x7f32541005e0 2024-03-28T18:49:41.646 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.642+0000 7f324bfff640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 0x7f32400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.646 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.642+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f32440f09a0 con 0x7f32541005e0 2024-03-28T18:49:41.646 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.642+0000 7f3249d9b640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 0x7f32400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.641+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32541a21d0 con 0x7f32541005e0 2024-03-28T18:49:41.650 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.650+0000 7f3249d9b640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 0x7f32400a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f323c009bd0 tx=0x7f323c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:41.655 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.655+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f32440bcfe0 con 0x7f32541005e0 2024-03-28T18:49:41.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:41 smithi192 ceph-mon[36589]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 645 MiB/s wr, 759 op/s 2024-03-28T18:49:41.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:41 smithi164 ceph-mon[36248]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 645 MiB/s wr, 759 op/s 2024-03-28T18:49:41.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:41 smithi145 ceph-mon[32151]: pgmap v708: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 645 MiB/s wr, 759 op/s 2024-03-28T18:49:41.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.853+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32541a2440 con 0x7f32541005e0 2024-03-28T18:49:41.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.855+0000 7f324bfff640 1 -- 172.21.15.145:0/3884319453 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7f32440c1080 con 0x7f32541005e0 2024-03-28T18:49:41.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:49:41.856 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:49:41.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 msgr2=0x7f32400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 0x7f32400a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f323c009bd0 tx=0x7f323c005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 msgr2=0x7f3254199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f3244004960 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 shutdown_connections 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f32400a2800 0x7f32400a4cc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3254102190 0x7f32541a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32541017d0 0x7f325419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 --2- 172.21.15.145:0/3884319453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32541005e0 0x7f3254199c70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.859 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 >> 172.21.15.145:0/3884319453 conn(0x7f32540fbb50 msgr2=0x7f32540fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:41.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.857+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 shutdown_connections 2024-03-28T18:49:41.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.858+0000 7f3258922640 1 -- 172.21.15.145:0/3884319453 wait complete. 2024-03-28T18:49:41.875 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:49:41.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- 172.21.15.145:0/2872060477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4104a70 msgr2=0x7f13f4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/2872060477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4104a70 0x7f13f4106e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009900 tx=0x7f13e402bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- 172.21.15.145:0/2872060477 shutdown_connections 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/2872060477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f41073a0 0x7f13f41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/2872060477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4104a70 0x7f13f4106e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.986 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/2872060477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f4102140 0x7f13f4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- 172.21.15.145:0/2872060477 >> 172.21.15.145:0/2872060477 conn(0x7f13f40fbb50 msgr2=0x7f13f40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- 172.21.15.145:0/2872060477 shutdown_connections 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- 172.21.15.145:0/2872060477 wait complete. 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 Processor -- start 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.983+0000 7f13fbbb9640 1 -- start start 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4102140 0x7f13f4101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f41073a0 0x7f13f4100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f410d330 con 0x7f13f4104a70 2024-03-28T18:49:41.987 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f410d1b0 con 0x7f13f4102140 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f410d4b0 con 0x7f13f41073a0 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:60250/0 (socket says 172.21.15.145:60250) 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 -- 172.21.15.145:0/1092492165 learned_addr learned my addr 172.21.15.145:0/1092492165 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9955640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4102140 0x7f13f4101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fa156640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f41073a0 0x7f13f4100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 -- 172.21.15.145:0/1092492165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f41073a0 msgr2=0x7f13f4100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f41073a0 0x7f13f4100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 -- 172.21.15.145:0/1092492165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4102140 msgr2=0x7f13f4101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:41.988 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4102140 0x7f13f4101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 -- 172.21.15.145:0/1092492165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4100ec0 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13f9154640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4041070 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f41a1ac0 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13e4004450 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4003df0 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.984+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f41a2060 con 0x7f13f4104a70 2024-03-28T18:49:41.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.986+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f40fd640 con 0x7f13f4104a70 2024-03-28T18:49:41.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.986+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f13e4044070 con 0x7f13f4104a70 2024-03-28T18:49:41.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.987+0000 7f13eaffd640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 0x7f13e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:41.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.987+0000 7f13f9955640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 0x7f13e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:41.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.987+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f13e40f08d0 con 0x7f13f4104a70 2024-03-28T18:49:41.990 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.988+0000 7f13f9955640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 0x7f13e00a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009bd0 tx=0x7f13f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:41.992 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:41.992+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13e40bd650 con 0x7f13f4104a70 2024-03-28T18:49:42.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.186+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13f402d070 con 0x7f13f4104a70 2024-03-28T18:49:42.187 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.187+0000 7f13eaffd640 1 -- 172.21.15.145:0/1092492165 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13e40bcf50 con 0x7f13f4104a70 2024-03-28T18:49:42.187 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:49:42.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 msgr2=0x7f13e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 0x7f13e00a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009bd0 tx=0x7f13f0005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 msgr2=0x7f13f41000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e402c640 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 shutdown_connections 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f13e00a2800 0x7f13e00a4cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13f41073a0 0x7f13f4100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13f4104a70 0x7f13f41000f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 --2- 172.21.15.145:0/1092492165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f4102140 0x7f13f4101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 >> 172.21.15.145:0/1092492165 conn(0x7f13f40fbb50 msgr2=0x7f13f4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 shutdown_connections 2024-03-28T18:49:42.190 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.189+0000 7f13fbbb9640 1 -- 172.21.15.145:0/1092492165 wait complete. 2024-03-28T18:49:42.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:49:42.312 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 -- 172.21.15.145:0/932731200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 msgr2=0x7f00c01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 --2- 172.21.15.145:0/932731200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c01089e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009ca0 tx=0x7f00bc02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 -- 172.21.15.145:0/932731200 shutdown_connections 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 --2- 172.21.15.145:0/932731200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c01089e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 --2- 172.21.15.145:0/932731200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 --2- 172.21.15.145:0/932731200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 -- 172.21.15.145:0/932731200 >> 172.21.15.145:0/932731200 conn(0x7f00c00fbb30 msgr2=0x7f00c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 -- 172.21.15.145:0/932731200 shutdown_connections 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.309+0000 7f00c7cff640 1 -- 172.21.15.145:0/932731200 wait complete. 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 Processor -- start 2024-03-28T18:49:42.313 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 -- start start 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c010c230 con 0x7f00c0102150 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c010c0b0 con 0x7f00c01005a0 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.310+0000 7f00c7cff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c010c3b0 con 0x7f00c0101790 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c5a9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c629c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.314 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c5a9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c0069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:37244/0 (socket says 172.21.15.145:37244) 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c5a9b640 1 -- 172.21.15.145:0/2976733029 learned_addr learned my addr 172.21.15.145:0/2976733029 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 -- 172.21.15.145:0/2976733029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 msgr2=0x7f00c0069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c0069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 -- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 msgr2=0x7f00c019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 -- 172.21.15.145:0/2976733029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c019f5e0 con 0x7f00c0101790 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c629c640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.311+0000 7f00c529a640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c019c6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00b000e9f0 tx=0x7f00b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.312+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b000ccc0 con 0x7f00c0101790 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.312+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00b0010910 con 0x7f00c0101790 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.312+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c019f870 con 0x7f00c0101790 2024-03-28T18:49:42.315 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.312+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b0018960 con 0x7f00c0101790 2024-03-28T18:49:42.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.312+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c019fdc0 con 0x7f00c0101790 2024-03-28T18:49:42.316 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.313+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c01078c0 con 0x7f00c0101790 2024-03-28T18:49:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.315+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f00b0010430 con 0x7f00c0101790 2024-03-28T18:49:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.315+0000 7f00b6ffd640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 0x7f00ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.315+0000 7f00c5a9b640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 0x7f00ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.315+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f00b00cf1f0 con 0x7f00c0101790 2024-03-28T18:49:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.316+0000 7f00c5a9b640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 0x7f00ac0a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f00b80058c0 tx=0x7f00b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.323 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.323+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f00b009b890 con 0x7f00c0101790 2024-03-28T18:49:42.500 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.500+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00c00622a0 con 0x7f00c0101790 2024-03-28T18:49:42.501 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.501+0000 7f00b6ffd640 1 -- 172.21.15.145:0/2976733029 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00b0084e20 con 0x7f00c0101790 2024-03-28T18:49:42.501 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:49:42.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 msgr2=0x7f00ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 0x7f00ac0a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f00b80058c0 tx=0x7f00b8005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 msgr2=0x7f00c019c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c019c6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00b000e9f0 tx=0x7f00b000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 shutdown_connections 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c0102150 0x7f00c019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f00ac0a2800 0x7f00ac0a4cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00c0101790 0x7f00c019c6b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 --2- 172.21.15.145:0/2976733029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00c01005a0 0x7f00c0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 >> 172.21.15.145:0/2976733029 conn(0x7f00c00fbb30 msgr2=0x7f00c0104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 shutdown_connections 2024-03-28T18:49:42.504 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.503+0000 7f00c7cff640 1 -- 172.21.15.145:0/2976733029 wait complete. 2024-03-28T18:49:42.514 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:42 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3884319453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:49:42.514 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:42 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1092492165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:49:42.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 -- 172.21.15.145:0/1623797124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a0101790 msgr2=0x7f72a0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1623797124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a0101790 0x7f72a0101c10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7290009900 tx=0x7f729002bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 -- 172.21.15.145:0/1623797124 shutdown_connections 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1623797124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0102150 0x7f72a01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1623797124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a0101790 0x7f72a0101c10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1623797124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a01005a0 0x7f72a01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 -- 172.21.15.145:0/1623797124 >> 172.21.15.145:0/1623797124 conn(0x7f72a00fbb30 msgr2=0x7f72a00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 -- 172.21.15.145:0/1623797124 shutdown_connections 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.648+0000 7f72a7c97640 1 -- 172.21.15.145:0/1623797124 wait complete. 2024-03-28T18:49:42.651 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 Processor -- start 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 -- start start 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0101790 0x7f72a0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a0102150 0x7f72a019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a010c230 con 0x7f72a01005a0 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a010c0b0 con 0x7f72a0101790 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a7c97640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a010c3b0 con 0x7f72a0102150 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a5a33640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a5a33640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:54978/0 (socket says 172.21.15.145:54978) 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a5a33640 1 -- 172.21.15.145:0/1129053253 learned_addr learned my addr 172.21.15.145:0/1129053253 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a6234640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a0102150 0x7f72a019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.652 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.649+0000 7f72a5232640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0101790 0x7f72a0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 -- 172.21.15.145:0/1129053253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a0102150 msgr2=0x7f72a019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a0102150 0x7f72a019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 -- 172.21.15.145:0/1129053253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0101790 msgr2=0x7f72a0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0101790 0x7f72a0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 -- 172.21.15.145:0/1129053253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a019f5e0 con 0x7f72a01005a0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a5a33640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f729c00e9c0 tx=0x7f729c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c00cd80 con 0x7f72a01005a0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f729c0035f0 con 0x7f72a01005a0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a019f8d0 con 0x7f72a01005a0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c010690 con 0x7f72a01005a0 2024-03-28T18:49:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.650+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a019fda0 con 0x7f72a01005a0 2024-03-28T18:49:42.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.651+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a01a0000 con 0x7f72a01005a0 2024-03-28T18:49:42.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.652+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f729c0037a0 con 0x7f72a01005a0 2024-03-28T18:49:42.654 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.652+0000 7f7296ffd640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.655 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.653+0000 7f72a5232640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.655 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.653+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f729c0cf170 con 0x7f72a01005a0 2024-03-28T18:49:42.655 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.653+0000 7f72a5232640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 0x7f728c0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f729002c7b0 tx=0x7f7290005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:42 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3884319453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:49:42.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:42 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1092492165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:49:42.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.657+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f729c09be60 con 0x7f72a01005a0 2024-03-28T18:49:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:42 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3884319453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:49:42.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:42 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1092492165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:49:42.849 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.848+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72a01a01d0 con 0x7f72a01005a0 2024-03-28T18:49:42.849 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.849+0000 7f7296ffd640 1 -- 172.21.15.145:0/1129053253 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f729c09b760 con 0x7f72a01005a0 2024-03-28T18:49:42.849 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 msgr2=0x7f728c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 0x7f728c0a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f729002c7b0 tx=0x7f7290005fd0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 msgr2=0x7f72a0197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f729c00e9c0 tx=0x7f729c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 shutdown_connections 2024-03-28T18:49:42.852 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f728c0a2800 0x7f728c0a4cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f72a0102150 0x7f72a019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72a0101790 0x7f72a0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 --2- 172.21.15.145:0/1129053253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72a01005a0 0x7f72a0197a30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 >> 172.21.15.145:0/1129053253 conn(0x7f72a00fbb30 msgr2=0x7f72a00fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.851+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 shutdown_connections 2024-03-28T18:49:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.852+0000 7f72a7c97640 1 -- 172.21.15.145:0/1129053253 wait complete. 2024-03-28T18:49:42.863 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.971+0000 7f02d6263640 1 -- 172.21.15.145:0/99283152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d0102160 msgr2=0x7f02d01089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.971+0000 7f02d6263640 1 --2- 172.21.15.145:0/99283152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d0102160 0x7f02d01089f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009ca0 tx=0x7f02c802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 -- 172.21.15.145:0/99283152 shutdown_connections 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 --2- 172.21.15.145:0/99283152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d0102160 0x7f02d01089f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 --2- 172.21.15.145:0/99283152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 --2- 172.21.15.145:0/99283152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d01005b0 0x7f02d01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 -- 172.21.15.145:0/99283152 >> 172.21.15.145:0/99283152 conn(0x7f02d00fbb40 msgr2=0x7f02d00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 -- 172.21.15.145:0/99283152 shutdown_connections 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 -- 172.21.15.145:0/99283152 wait complete. 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 Processor -- start 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.972+0000 7f02d6263640 1 -- start start 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 0x7f02d019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d010c230 con 0x7f02d01005b0 2024-03-28T18:49:42.976 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d010c0b0 con 0x7f02d0102160 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d6263640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d010c3b0 con 0x7f02d01017a0 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:55002/0 (socket says 172.21.15.145:55002) 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d4800640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 0x7f02d019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cffff640 1 -- 172.21.15.145:0/3316332837 learned_addr learned my addr 172.21.15.145:0/3316332837 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 -- 172.21.15.145:0/3316332837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 msgr2=0x7f02d019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 0x7f02d019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 -- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 msgr2=0x7f02d0197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 -- 172.21.15.145:0/3316332837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d019f5b0 con 0x7f02d01017a0 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02d4800640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 0x7f02d019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cffff640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cf7fe640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0197ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e9f0 tx=0x7f02c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.973+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c400ccc0 con 0x7f02d01017a0 2024-03-28T18:49:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.974+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02c4005020 con 0x7f02d01017a0 2024-03-28T18:49:42.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.974+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d019f8a0 con 0x7f02d01017a0 2024-03-28T18:49:42.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.974+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40049e0 con 0x7f02d01017a0 2024-03-28T18:49:42.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.974+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d019fda0 con 0x7f02d01017a0 2024-03-28T18:49:42.979 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.975+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d01a0000 con 0x7f02d01017a0 2024-03-28T18:49:42.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.976+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f02c4005560 con 0x7f02d01017a0 2024-03-28T18:49:42.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.976+0000 7f02cd7fa640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 0x7f02b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:42.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.976+0000 7f02cffff640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 0x7f02b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:42.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.976+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f02c40cedc0 con 0x7f02d01017a0 2024-03-28T18:49:42.980 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.977+0000 7f02cffff640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 0x7f02b80a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f02c00058c0 tx=0x7f02c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:42.981 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:42.981+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f02c40051d0 con 0x7f02d01017a0 2024-03-28T18:49:43.177 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.176+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f02d01a01d0 con 0x7f02d01017a0 2024-03-28T18:49:43.177 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.177+0000 7f02cd7fa640 1 -- 172.21.15.145:0/3316332837 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7f02c409bab0 con 0x7f02d01017a0 2024-03-28T18:49:43.177 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 msgr2=0x7f02b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 0x7f02b80a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f02c00058c0 tx=0x7f02c0005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 msgr2=0x7f02d0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0197ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e9f0 tx=0x7f02c400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 shutdown_connections 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02d0102160 0x7f02d019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f02b80a2800 0x7f02b80a4cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02d01017a0 0x7f02d0197ed0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.180 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 --2- 172.21.15.145:0/3316332837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d01005b0 0x7f02d0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 >> 172.21.15.145:0/3316332837 conn(0x7f02d00fbb40 msgr2=0x7f02d00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:43.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 shutdown_connections 2024-03-28T18:49:43.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.179+0000 7f02d6263640 1 -- 172.21.15.145:0/3316332837 wait complete. 2024-03-28T18:49:43.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T18:49:43.302 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- 172.21.15.145:0/971903336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 msgr2=0x7fa09c109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:43.302 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/971903336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c109b30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa094009ca0 tx=0x7fa09402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.302 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- 172.21.15.145:0/971903336 shutdown_connections 2024-03-28T18:49:43.302 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/971903336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c109b30 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.302 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/971903336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1028e0 0x7fa09c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/971903336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1016f0 0x7fa09c101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- 172.21.15.145:0/971903336 >> 172.21.15.145:0/971903336 conn(0x7fa09c0fcc40 msgr2=0x7fa09c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- 172.21.15.145:0/971903336 shutdown_connections 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- 172.21.15.145:0/971903336 wait complete. 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 Processor -- start 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- start start 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 0x7fa09c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 0x7fa09c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:43.303 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c10d370 con 0x7fa09c1032a0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c10d1f0 con 0x7fa09c1016f0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.299+0000 7fa0a2d77640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c10d4f0 con 0x7fa09c1028e0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a0b13640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 0x7fa09c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a1314640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 0x7fa09c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a0b13640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 0x7fa09c197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:47368/0 (socket says 172.21.15.145:47368) 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a0b13640 1 -- 172.21.15.145:0/1208348310 learned_addr learned my addr 172.21.15.145:0/1208348310 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a1314640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:55008/0 (socket says 172.21.15.145:55008) 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 -- 172.21.15.145:0/1208348310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 msgr2=0x7fa09c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 0x7fa09c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 -- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 msgr2=0x7fa09c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 -- 172.21.15.145:0/1208348310 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c19f5a0 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa093fff640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 0x7fa09c197ec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9f0 tx=0x7fa08c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00cd80 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c19f890 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa08c0035f0 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c010670 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.300+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c19fe30 con 0x7fa09c1028e0 2024-03-28T18:49:43.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.301+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c108a10 con 0x7fa09c1028e0 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.302+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa08c0037a0 con 0x7fa09c1028e0 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.303+0000 7fa091ffb640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 0x7fa0840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.303+0000 7fa0a0b13640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 0x7fa0840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.303+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa08c0cf090 con 0x7fa09c1028e0 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.303+0000 7fa0a0b13640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 0x7fa0840a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0980058c0 tx=0x7fa098005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:43.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.306+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa08c09bd20 con 0x7fa09c1028e0 2024-03-28T18:49:43.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:43 smithi145 ceph-mon[32151]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 571 MiB/s wr, 674 op/s 2024-03-28T18:49:43.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:43 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2976733029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:49:43.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:43 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1129053253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:49:43.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:43 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3316332837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:49:43.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.549+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fa09c02d1f0 con 0x7fa09c1028e0 2024-03-28T18:49:43.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.558+0000 7fa091ffb640 1 -- 172.21.15.145:0/1208348310 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+739833 (secure 0 0 0) 0x7fa08c09b620 con 0x7fa09c1028e0 2024-03-28T18:49:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:49:43.573 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:49:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "epoch": 20, 2024-03-28T18:49:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "active_gid": 14152, 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "active_name": "x", 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "active_addrs": { 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "addrvec": [ 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v2", 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:6800", 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 82375532 2024-03-28T18:49:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:6801", 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 82375532 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "active_addr": "172.21.15.145:6801/82375532", 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "active_change": "2024-03-28T18:28:53.394169+0000", 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T18:49:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "available": true, 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "standbys": [ 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "gid": 24109, 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "z", 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr_features": 4540701547738038271, 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "available_modules": [ 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "alerts", 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_destination": { 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_destination", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_from_name": { 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_from_name", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "Ceph", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email From: name", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_host": { 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_host", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP server", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_password": { 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_password", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Password to authenticate with", 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_port": { 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_port", 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "465", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP port", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_sender": { 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_sender", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_ssl": { 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_ssl", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:49:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_user": { 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_user", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "User to authenticate as", 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "balancer", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "active": { 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "active", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_time": { 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_time", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0000", 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_weekday": { 2024-03-28T18:49:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_weekday", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_max_iterations": { 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "25", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "250", 2024-03-28T18:49:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_metrics": { 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_metrics", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:49:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_step": { 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_step", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.5", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0.001", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "0.999", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "end_time": { 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_time", 2024-03-28T18:49:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2359", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "end_weekday": { 2024-03-28T18:49:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_weekday", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "min_score": { 2024-03-28T18:49:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_score", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "upmap", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "crush-compat", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "none", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "read", 2024-03-28T18:49:43.613 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap-read" 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Balancer mode", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_ids": { 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_ids", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.614 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.615 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_deviation": { 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_deviation", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.616 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.617 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_optimizations": { 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.618 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm", 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_down_multiplier": { 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_down_multiplier", 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.619 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_refresh_rate": { 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_refresh_rate", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.620 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "20", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_starting_port": { 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_starting_port", 2024-03-28T18:49:43.621 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4721", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.622 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_ptrace": { 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_ptrace", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_interval": { 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_interval", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.7", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.625 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm_log_destination": { 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm_log_destination", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "file", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "file,syslog", 2024-03-28T18:49:43.626 INFO:teuthology.orchestra.run.smithi145.stdout: "syslog" 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "cgroups_split": { 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cgroups_split", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.627 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "config_checks_enabled": { 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_checks_enabled", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.628 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "config_dashboard": { 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_dashboard", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.629 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_alertmanager": { 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_alertmanager", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.630 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_base": { 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_base", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.631 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_elasticsearch": { 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.632 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "elasticsearch container image", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_grafana": { 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_grafana", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.633 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_haproxy": { 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_haproxy", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.634 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "HAproxy container image", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_agent": { 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:49:43.635 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_collector": { 2024-03-28T18:49:43.636 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_query": { 2024-03-28T18:49:43.637 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger query container image", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.638 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_keepalived": { 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_keepalived", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Keepalived container image", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_loki": { 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_loki", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Loki container image", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_node_exporter": { 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_node_exporter", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_nvmeof": { 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_nvmeof", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Nvme-of container image", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_prometheus": { 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_prometheus", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_promtail": { 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_promtail", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Promtail container image", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.644 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_snmp_gateway": { 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.645 INFO:teuthology.orchestra.run.smithi145.stdout: "container_init": { 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_init", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.646 INFO:teuthology.orchestra.run.smithi145.stdout: "daemon_cache_timeout": { 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.647 INFO:teuthology.orchestra.run.smithi145.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "900", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.648 INFO:teuthology.orchestra.run.smithi145.stdout: "default_registry": { 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_registry", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "device_cache_timeout": { 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_cache_timeout", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1800", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.650 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "device_enhanced_scan": { 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_enhanced_scan", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "facts_cache_timeout": { 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "facts_cache_timeout", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "host_check_interval": { 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "host_check_interval", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "hw_monitoring": { 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hw_monitoring", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.654 INFO:teuthology.orchestra.run.smithi145.stdout: "inventory_list_all": { 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "inventory_list_all", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.655 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.656 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "log_refresh_metadata": { 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_refresh_metadata", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.657 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.658 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.659 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.660 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.662 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "*", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "max_count_per_host": { 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_count_per_host", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.664 INFO:teuthology.orchestra.run.smithi145.stdout: "max_osd_draining_count": { 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_osd_draining_count", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "migration_current": { 2024-03-28T18:49:43.665 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "migration_current", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "internal - do not modify", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:43.666 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "root", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm-package", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "root" 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "oob_default_addr": { 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "oob_default_addr", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "169.254.1.1", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.668 INFO:teuthology.orchestra.run.smithi145.stdout: "prometheus_alerts_path": { 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_insecure": { 2024-03-28T18:49:43.669 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_insecure", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_password": { 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_password", 2024-03-28T18:49:43.670 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_url": { 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_url", 2024-03-28T18:49:43.671 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_username": { 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_username", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.672 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "secure_monitoring_stack": { 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.673 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "service_discovery_port": { 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "service_discovery_port", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.674 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8765", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "ssh_config_file": { 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssh_config_file", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.675 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "use_agent": { 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_agent", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.676 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "use_repo_digest": { 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_repo_digest", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.677 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_failed_host_check": { 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.678 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_daemons": { 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.679 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_hosts": { 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.680 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crash", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.681 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.682 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.683 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.684 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.685 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "retain_interval": { 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retain_interval", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "31536000", 2024-03-28T18:49:43.686 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_recent_interval": { 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_recent_interval", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1209600", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.687 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dashboard", 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:49:43.688 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.689 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:49:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:49:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.702 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_URL": { 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.703 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:43.704 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.705 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.706 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.707 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "{}", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:49:43.709 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.710 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:49:43.711 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.712 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:49:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:49:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "45", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:49:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "cross_origin_url": { 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cross_origin_url", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "crt_file": { 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crt_file", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "debug": { 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "debug", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "jwt_token_ttl": { 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "jwt_token_ttl", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "28800", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "motd": { 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "motd", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "The message of the day", 2024-03-28T18:49:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8080", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl_server_port": { 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl_server_port", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8443", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "redirect", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect" 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "url_prefix": { 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url_prefix", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "devicehealth", 2024-03-28T18:49:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_monitoring": { 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_monitoring", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "monitor device health metrics", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "mark_out_threshold": { 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mark_out_threshold", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2419200", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_name": { 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_name", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "device_health_metrics", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "retention_period": { 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retention_period", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15552000", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:49:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_frequency": { 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_frequency", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "self_heal": { 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "self_heal", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_threshold": { 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_threshold", 2024-03-28T18:49:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7257600", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "diskprediction_local", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "predict_interval": { 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predict_interval", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "predictor_model": { 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predictor_model", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "prophetstor", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "influx", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": false, 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "batch_size": { 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "batch_size", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5000", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "database": { 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "database", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "ceph", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": { 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hostname", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "30", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "5", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "password": { 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "password", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "port": { 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "port", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8086", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server port", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "false", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "threads": { 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threads", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "32", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "username": { 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "username", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "verify_ssl": { 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "verify_ssl", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "true", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "insights", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "iostat", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "k8sevents", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_event_retention_days": { 2024-03-28T18:49:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "config_check_secs": { 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_check_secs", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "10", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "localpool", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "failure_domain": { 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "failure_domain", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "host", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "min_size": { 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_size", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "num_rep": { 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "num_rep", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_num": { 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_num", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "128", 2024-03-28T18:49:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "prefix": { 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prefix", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "subtree": { 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "subtree", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "rack", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:49:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mds_autoscaler", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirroring", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "nfs", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "fail_fs": { 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "fail_fs", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator": { 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "rook", 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "test_orchestrator" 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Orchestrator backend", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_perf_query", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_support", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_autoscaler", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "threshold": { 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threshold", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1.0", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "scaling threshold", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "progress", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_pg_recovery_event": { 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "max_completed_events": { 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_completed_events", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "50", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "cache": { 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cache", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "exclude_perf_counters": { 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "exclude_perf_counters", 2024-03-28T18:49:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools": { 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "300", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_interval": { 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_interval", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15.0", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "9283", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "stale_cache_strategy": { 2024-03-28T18:49:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stale_cache_strategy", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "log", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "default", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "default", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "error" 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_support", 2024-03-28T18:49:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_snap_create": { 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "trash_purge_schedule": { 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "trash_purge_schedule", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "restful", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_auth": { 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_auth", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rgw", 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rook", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "storage_class": { 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "storage_class", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.891 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "local", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "selftest", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.892 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "roption1": { 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption1", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "roption2": { 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption2", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "xyz", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption1": { 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption1", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption2": { 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption2", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption3": { 2024-03-28T18:49:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption3", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption4": { 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption4", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption5": { 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption5", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption6": { 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption6", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption7": { 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption7", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "42", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "testkey": { 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testkey", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "testlkey": { 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testlkey", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "testnewline": { 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testnewline", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snap_schedule", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_m_granularity": { 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_m_granularity", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "dump_on_update": { 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dump_on_update", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stats", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "status", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telegraf", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "address": { 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "address", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telemetry", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_basic": { 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_basic", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_crash": { 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_crash", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:49:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_device": { 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_device", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_ident": { 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_ident", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_perf": { 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_perf", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "contact": { 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "contact", 2024-03-28T18:49:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "description": { 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "description", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "device_url": { 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_url", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "24", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "8", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "last_opt_revision": { 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "last_opt_revision", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard": { 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard_description": { 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard_description", 2024-03-28T18:49:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "organization": { 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "organization", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "proxy": { 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "proxy", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "url": { 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "test_orchestrator", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "volumes", 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_clones": { 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_clones", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "periodic_async_work": { 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "periodic_async_work", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Periodically check for async work", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_delay": { 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "discovery_interval": { 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "discovery_interval", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "100", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "identifier": { 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "identifier", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_host": { 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_host", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_port": { 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_port", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10051", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_sender": { 2024-03-28T18:49:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_sender", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "gid": 24110, 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "y", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "mgr_features": 4540701547738038271, 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "available_modules": [ 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "alerts", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.978 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_destination": { 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_destination", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.979 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_from_name": { 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_from_name", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "Ceph", 2024-03-28T18:49:43.980 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email From: name", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_host": { 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_host", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.981 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP server", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_password": { 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_password", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Password to authenticate with", 2024-03-28T18:49:43.982 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_port": { 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_port", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "465", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP port", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_sender": { 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_sender", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.984 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_ssl": { 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_ssl", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_user": { 2024-03-28T18:49:43.985 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_user", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "User to authenticate as", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:43.986 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "balancer", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "active": { 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "active", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.987 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_time": { 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_time", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0000", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_weekday": { 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_weekday", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_max_iterations": { 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:49:43.989 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "25", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "250", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_metrics": { 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_metrics", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.990 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_step": { 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_step", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.5", 2024-03-28T18:49:43.991 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0.001", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "0.999", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "end_time": { 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_time", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2359", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.992 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "end_weekday": { 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_weekday", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:43.993 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.994 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.995 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:43.996 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:43.997 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "min_score": { 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_score", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.998 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "upmap", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:43.999 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "crush-compat", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "none", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "read", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap-read" 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Balancer mode", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_ids": { 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_ids", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.000 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.001 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_deviation": { 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_deviation", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.002 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_optimizations": { 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.003 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_down_multiplier": { 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_down_multiplier", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.004 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_refresh_rate": { 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_refresh_rate", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "20", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:49:44.005 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_starting_port": { 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_starting_port", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4721", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.006 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_ptrace": { 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_ptrace", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.007 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_interval": { 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_interval", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.008 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.7", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.009 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm_log_destination": { 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm_log_destination", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "file", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "file,syslog", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "syslog" 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:49:44.010 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "cgroups_split": { 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cgroups_split", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.011 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "config_checks_enabled": { 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_checks_enabled", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.012 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "config_dashboard": { 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_dashboard", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.013 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_alertmanager": { 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_alertmanager", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.014 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_base": { 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_base", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.015 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_elasticsearch": { 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "elasticsearch container image", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.016 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_grafana": { 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_grafana", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_haproxy": { 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_haproxy", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "HAproxy container image", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_agent": { 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_collector": { 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_query": { 2024-03-28T18:49:44.020 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger query container image", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_keepalived": { 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_keepalived", 2024-03-28T18:49:44.021 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Keepalived container image", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_loki": { 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_loki", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.022 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Loki container image", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_node_exporter": { 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_node_exporter", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.023 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_nvmeof": { 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_nvmeof", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:49:44.024 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Nvme-of container image", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_prometheus": { 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_prometheus", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.025 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_promtail": { 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_promtail", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Promtail container image", 2024-03-28T18:49:44.026 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_snmp_gateway": { 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.027 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "container_init": { 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_init", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.028 INFO:teuthology.orchestra.run.smithi145.stdout: "daemon_cache_timeout": { 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:49:44.029 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "900", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "default_registry": { 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_registry", 2024-03-28T18:49:44.030 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "device_cache_timeout": { 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_cache_timeout", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.031 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1800", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "device_enhanced_scan": { 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_enhanced_scan", 2024-03-28T18:49:44.032 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "facts_cache_timeout": { 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "facts_cache_timeout", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.033 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "host_check_interval": { 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "host_check_interval", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.034 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "hw_monitoring": { 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hw_monitoring", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.035 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "inventory_list_all": { 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "inventory_list_all", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.036 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "log_refresh_metadata": { 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_refresh_metadata", 2024-03-28T18:49:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.039 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.040 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.041 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.042 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.043 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "*", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "max_count_per_host": { 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_count_per_host", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.045 INFO:teuthology.orchestra.run.smithi145.stdout: "max_osd_draining_count": { 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_osd_draining_count", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "migration_current": { 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "migration_current", 2024-03-28T18:49:44.046 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "internal - do not modify", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "root", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm-package", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "root" 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "oob_default_addr": { 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "oob_default_addr", 2024-03-28T18:49:44.048 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "169.254.1.1", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "prometheus_alerts_path": { 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.049 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_insecure": { 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_insecure", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.050 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_password": { 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_password", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.051 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_url": { 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_url", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.052 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_username": { 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_username", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.053 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "secure_monitoring_stack": { 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "service_discovery_port": { 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "service_discovery_port", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8765", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "ssh_config_file": { 2024-03-28T18:49:44.055 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssh_config_file", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "use_agent": { 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_agent", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.056 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "use_repo_digest": { 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_repo_digest", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.057 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_failed_host_check": { 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.058 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_daemons": { 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.059 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_hosts": { 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.060 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crash", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.061 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.062 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.063 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.064 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "retain_interval": { 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retain_interval", 2024-03-28T18:49:44.065 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "31536000", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_recent_interval": { 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_recent_interval", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.066 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1209600", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dashboard", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.067 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:49:44.068 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.069 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.070 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.071 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.072 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:49:44.073 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:49:44.074 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.075 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.076 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.077 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.078 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.080 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.081 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_URL": { 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:49:44.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.083 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.084 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.085 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.086 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "{}", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.088 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:49:44.089 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.091 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.092 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.093 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.094 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:49:44.095 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.096 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.097 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8", 2024-03-28T18:49:44.098 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "45", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.099 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.100 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.101 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:49:44.103 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.104 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.105 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.106 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "cross_origin_url": { 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cross_origin_url", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.107 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "crt_file": { 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crt_file", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.108 INFO:teuthology.orchestra.run.smithi145.stdout: "debug": { 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "debug", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "jwt_token_ttl": { 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "jwt_token_ttl", 2024-03-28T18:49:44.109 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "28800", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.110 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.111 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.112 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.113 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.114 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.115 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "motd": { 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "motd", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "The message of the day", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.117 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.118 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8080", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.119 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl_server_port": { 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl_server_port", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8443", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.120 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "redirect", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.121 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect" 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.122 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "url_prefix": { 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url_prefix", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "devicehealth", 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_monitoring": { 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_monitoring", 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "monitor device health metrics", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.125 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.126 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.129 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.130 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "mark_out_threshold": { 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mark_out_threshold", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2419200", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.131 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_name": { 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_name", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "device_health_metrics", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.132 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "retention_period": { 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retention_period", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15552000", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.133 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_frequency": { 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_frequency", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "self_heal": { 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "self_heal", 2024-03-28T18:49:44.134 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_threshold": { 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_threshold", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7257600", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "diskprediction_local", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.137 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.139 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.140 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.141 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "predict_interval": { 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predict_interval", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.142 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "predictor_model": { 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predictor_model", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "prophetstor", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.143 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.144 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "influx", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": false, 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "batch_size": { 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "batch_size", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5000", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.145 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "database": { 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "database", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "ceph", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.146 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": { 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hostname", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.147 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "30", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "5", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.149 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.150 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.151 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.152 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "password": { 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "password", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.153 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "port": { 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "port", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8086", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.154 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server port", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "false", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.155 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "threads": { 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threads", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "32", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:49:44.156 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "username": { 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "username", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:49:44.157 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "verify_ssl": { 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "verify_ssl", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "true", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.158 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "insights", 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.159 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.160 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.162 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.163 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "iostat", 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.164 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.165 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.166 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.167 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.168 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "k8sevents", 2024-03-28T18:49:44.169 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_event_retention_days": { 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.170 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "config_check_secs": { 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_check_secs", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "10", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.172 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.173 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.174 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.175 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.176 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "localpool", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "failure_domain": { 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "failure_domain", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "host", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:49:44.177 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.178 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.179 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.180 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.181 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "min_size": { 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_size", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.182 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "num_rep": { 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "num_rep", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.183 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_num": { 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_num", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "128", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:49:44.184 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "prefix": { 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prefix", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.185 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "subtree": { 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "subtree", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "rack", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.186 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mds_autoscaler", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.187 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.188 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.189 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.190 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.191 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirroring", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.192 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.193 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.194 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "nfs", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.197 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.198 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.199 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.200 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:44.201 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "fail_fs": { 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "fail_fs", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.203 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.204 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.205 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.206 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator": { 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm", 2024-03-28T18:49:44.207 INFO:teuthology.orchestra.run.smithi145.stdout: "rook", 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "test_orchestrator" 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Orchestrator backend", 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_perf_query", 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.208 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.209 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.210 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.211 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.212 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_support", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.213 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.214 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.215 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.216 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.217 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_autoscaler", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.218 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.219 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.220 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.221 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: "threshold": { 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threshold", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1.0", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "scaling threshold", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "progress", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_pg_recovery_event": { 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.225 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.227 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.228 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.229 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.230 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "max_completed_events": { 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_completed_events", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "50", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.232 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "cache": { 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cache", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.233 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "exclude_perf_counters": { 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "exclude_perf_counters", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.234 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.235 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.236 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.237 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.238 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools": { 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.239 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "300", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.240 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_interval": { 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_interval", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15.0", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.241 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.242 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "9283", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.243 INFO:teuthology.orchestra.run.smithi145.stdout: "stale_cache_strategy": { 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stale_cache_strategy", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "log", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:44.244 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "default", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "default", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "error" 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:44.246 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_support", 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.248 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.249 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.250 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.251 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.252 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.253 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.254 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.255 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.256 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_snap_create": { 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.257 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.258 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "trash_purge_schedule": { 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "trash_purge_schedule", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.259 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "restful", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_auth": { 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_auth", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.260 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.261 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.262 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.263 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.264 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.265 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.266 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.267 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rgw", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.268 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.270 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.271 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.272 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rook", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.273 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.274 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.275 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.276 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.277 INFO:teuthology.orchestra.run.smithi145.stdout: "storage_class": { 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "storage_class", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "local", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "selftest", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.279 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.280 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.281 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.282 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "roption1": { 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption1", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.283 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "roption2": { 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption2", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "xyz", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.284 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption1": { 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption1", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.285 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption2": { 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption2", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.286 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption3": { 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption3", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption4": { 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption4", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption5": { 2024-03-28T18:49:44.288 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption5", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption6": { 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption6", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.289 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption7": { 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption7", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.290 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "42", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "testkey": { 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testkey", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.291 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "testlkey": { 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testlkey", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.292 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "testnewline": { 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testnewline", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.293 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snap_schedule", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_m_granularity": { 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_m_granularity", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.294 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "dump_on_update": { 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dump_on_update", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.295 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.296 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.297 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.298 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.299 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stats", 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.300 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.301 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.302 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.303 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.304 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "status", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.305 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.306 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.307 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.308 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.309 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telegraf", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "address": { 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "address", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.310 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.311 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.312 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.313 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.314 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.315 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telemetry", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_basic": { 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_basic", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.317 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_crash": { 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_crash", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:49:44.318 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_device": { 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_device", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.319 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_ident": { 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_ident", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.320 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_perf": { 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_perf", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "contact": { 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "contact", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "description": { 2024-03-28T18:49:44.322 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "description", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "device_url": { 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_url", 2024-03-28T18:49:44.323 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.324 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.325 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "24", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "8", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "last_opt_revision": { 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "last_opt_revision", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.326 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard": { 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.327 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard_description": { 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard_description", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.328 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.329 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.330 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.331 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.332 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "organization": { 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "organization", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.333 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "proxy": { 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "proxy", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.334 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "url": { 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.335 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "test_orchestrator", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.336 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.337 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.338 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.339 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.340 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "volumes", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.341 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.342 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.343 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.344 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_clones": { 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_clones", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.345 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "periodic_async_work": { 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "periodic_async_work", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.346 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Periodically check for async work", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_delay": { 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.347 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.348 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "discovery_interval": { 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "discovery_interval", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.349 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "100", 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.350 INFO:teuthology.orchestra.run.smithi145.stdout: "identifier": { 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "identifier", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.354 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.355 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.356 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.357 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.358 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.359 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_host": { 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_host", 2024-03-28T18:49:44.360 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_port": { 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_port", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.361 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10051", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_sender": { 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_sender", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.362 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "modules": [ 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm", 2024-03-28T18:49:44.363 INFO:teuthology.orchestra.run.smithi145.stdout: "dashboard", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "iostat", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "nfs", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "restful" 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "available_modules": [ 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "alerts", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.364 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.365 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.366 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.367 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.368 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_destination": { 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_destination", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_from_name": { 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_from_name", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "Ceph", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Email From: name", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_host": { 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_host", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP server", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_password": { 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_password", 2024-03-28T18:49:44.372 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Password to authenticate with", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_port": { 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_port", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.373 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "465", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP port", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_sender": { 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_sender", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.374 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_ssl": { 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_ssl", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.375 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "smtp_user": { 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "smtp_user", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "User to authenticate as", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.376 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "balancer", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "active": { 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "active", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.377 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_time": { 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_time", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0000", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.378 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "begin_weekday": { 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "begin_weekday", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.379 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_max_iterations": { 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "25", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "250", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.380 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_metrics": { 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_metrics", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.381 INFO:teuthology.orchestra.run.smithi145.stdout: "crush_compat_step": { 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crush_compat_step", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.5", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0.001", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "0.999", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "end_time": { 2024-03-28T18:49:44.382 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_time", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2359", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "end_weekday": { 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "end_weekday", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.383 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "0", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "6", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.384 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.385 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.386 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.387 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "min_score": { 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_score", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:44.389 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "upmap", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "crush-compat", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "none", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "read", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap-read" 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Balancer mode", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.390 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_ids": { 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_ids", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.391 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_deviation": { 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_deviation", 2024-03-28T18:49:44.392 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "upmap_max_optimizations": { 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.393 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.394 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_down_multiplier": { 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_down_multiplier", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.395 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_refresh_rate": { 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_refresh_rate", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "20", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "agent_starting_port": { 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "agent_starting_port", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.396 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4721", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_ptrace": { 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_ptrace", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.397 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_interval": { 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_interval", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.398 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0.7", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.399 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm_log_destination": { 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cephadm_log_destination", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "file", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "file,syslog", 2024-03-28T18:49:44.400 INFO:teuthology.orchestra.run.smithi145.stdout: "syslog" 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "cgroups_split": { 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cgroups_split", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.401 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "config_checks_enabled": { 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_checks_enabled", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.402 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "config_dashboard": { 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_dashboard", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.403 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_alertmanager": { 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_alertmanager", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_base": { 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_base", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.405 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_elasticsearch": { 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "elasticsearch container image", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_grafana": { 2024-03-28T18:49:44.406 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_grafana", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_haproxy": { 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_haproxy", 2024-03-28T18:49:44.407 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "HAproxy container image", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_agent": { 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.408 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_collector": { 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.409 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_jaeger_query": { 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:49:44.410 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Jaeger query container image", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_keepalived": { 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_keepalived", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.411 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Keepalived container image", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_loki": { 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_loki", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.412 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Loki container image", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_node_exporter": { 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_node_exporter", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.413 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_nvmeof": { 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_nvmeof", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Nvme-of container image", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.414 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_prometheus": { 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_prometheus", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Prometheus container image", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_promtail": { 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_promtail", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Promtail container image", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "container_image_snmp_gateway": { 2024-03-28T18:49:44.416 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "container_init": { 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "container_init", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.417 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "daemon_cache_timeout": { 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.418 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.419 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "900", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "default_registry": { 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "default_registry", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.420 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "docker.io", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "device_cache_timeout": { 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_cache_timeout", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.421 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1800", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "device_enhanced_scan": { 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_enhanced_scan", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.422 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "facts_cache_timeout": { 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "facts_cache_timeout", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "host_check_interval": { 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "host_check_interval", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:49:44.424 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "hw_monitoring": { 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hw_monitoring", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "inventory_list_all": { 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "inventory_list_all", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.426 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.427 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "log_refresh_metadata": { 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_refresh_metadata", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.428 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.430 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.431 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:49:44.432 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "*", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.433 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "max_count_per_host": { 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_count_per_host", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.434 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "max_osd_draining_count": { 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_osd_draining_count", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.435 INFO:teuthology.orchestra.run.smithi145.stdout: "migration_current": { 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "migration_current", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "internal - do not modify", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "mode": { 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mode", 2024-03-28T18:49:44.436 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "root", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm-package", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "root" 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "oob_default_addr": { 2024-03-28T18:49:44.437 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "oob_default_addr", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "169.254.1.1", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "prometheus_alerts_path": { 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:49:44.438 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_insecure": { 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_insecure", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_password": { 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_password", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.440 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_url": { 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_url", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.441 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "registry_username": { 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "registry_username", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.442 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "secure_monitoring_stack": { 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.443 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "service_discovery_port": { 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "service_discovery_port", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8765", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "ssh_config_file": { 2024-03-28T18:49:44.444 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssh_config_file", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "use_agent": { 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_agent", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.445 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "use_repo_digest": { 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "use_repo_digest", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.446 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_failed_host_check": { 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.447 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_daemons": { 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.448 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_on_stray_hosts": { 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.449 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crash", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.450 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.452 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.453 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.454 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "retain_interval": { 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retain_interval", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "31536000", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.455 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_recent_interval": { 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_recent_interval", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1209600", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dashboard", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.457 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:49:44.459 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:49:44.460 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.461 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.462 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.463 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.464 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.465 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.466 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:49:44.468 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:49:44.469 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:49:44.470 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_URL": { 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.471 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.472 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.473 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.474 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.475 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "{}", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:49:44.477 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.478 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.479 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.480 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.481 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.482 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.483 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.484 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:49:44.485 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.486 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.487 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.488 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "45", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.489 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.490 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "admin", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.491 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.493 INFO:teuthology.orchestra.run.smithi145.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:49:44.494 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.495 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.496 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "cross_origin_url": { 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cross_origin_url", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.497 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "crt_file": { 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "crt_file", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.498 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "debug": { 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "debug", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.499 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "jwt_token_ttl": { 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "jwt_token_ttl", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "28800", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.500 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.502 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.503 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.504 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.505 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "motd": { 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "motd", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.506 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "The message of the day", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8080", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:44.509 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl_server_port": { 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl_server_port", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.510 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8443", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.511 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "redirect", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "redirect" 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:44.512 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "url_prefix": { 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url_prefix", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.513 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "devicehealth", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.514 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_monitoring": { 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_monitoring", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "monitor device health metrics", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.515 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.516 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.517 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.518 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.519 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "mark_out_threshold": { 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mark_out_threshold", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "2419200", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "pool_name": { 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pool_name", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "device_health_metrics", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.521 INFO:teuthology.orchestra.run.smithi145.stdout: "retention_period": { 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "retention_period", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15552000", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_frequency": { 2024-03-28T18:49:44.522 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_frequency", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "self_heal": { 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "self_heal", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.523 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.524 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "warn_threshold": { 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "warn_threshold", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7257600", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.525 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "diskprediction_local", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.526 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.528 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.529 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "predict_interval": { 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predict_interval", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "86400", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "predictor_model": { 2024-03-28T18:49:44.531 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "predictor_model", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "prophetstor", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "600", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "influx", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": false, 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "batch_size": { 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "batch_size", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5000", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.534 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "database": { 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "database", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "ceph", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.535 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": { 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "hostname", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.536 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "30", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "5", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.537 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.538 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.539 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.540 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.541 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "password": { 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "password", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.542 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "port": { 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "port", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "8086", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "InfluxDB server port", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.543 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "ssl": { 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ssl", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "false", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "threads": { 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threads", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "32", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "username": { 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "username", 2024-03-28T18:49:44.545 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "verify_ssl": { 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "verify_ssl", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.546 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "true", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "insights", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.547 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.548 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.549 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.550 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.551 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "iostat", 2024-03-28T18:49:44.552 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.553 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.554 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.555 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.555 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.557 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.558 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "k8sevents", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_event_retention_days": { 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "7", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:49:44.560 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "config_check_secs": { 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "config_check_secs", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "10", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.562 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.563 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.564 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.565 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "localpool", 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "failure_domain": { 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "failure_domain", 2024-03-28T18:49:44.566 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "host", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.567 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.568 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.569 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.570 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.571 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "min_size": { 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "min_size", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "num_rep": { 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "num_rep", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.573 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_num": { 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_num", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "128", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "prefix": { 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prefix", 2024-03-28T18:49:44.574 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "subtree": { 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "subtree", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.575 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "rack", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mds_autoscaler", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.576 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.577 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.578 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.579 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.580 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirroring", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.581 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.582 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.583 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.584 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.585 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "nfs", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.586 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.587 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.589 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.590 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "fail_fs": { 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "fail_fs", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.591 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.592 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.593 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.594 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.595 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator": { 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "orchestrator", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.596 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "cephadm", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "rook", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "test_orchestrator" 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Orchestrator backend", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.597 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_perf_query", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.598 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.599 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.601 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.602 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "osd_support", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.603 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.604 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.605 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.606 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "pg_autoscaler", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.608 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.609 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.610 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.611 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.612 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "threshold": { 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "threshold", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.613 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "3.0", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1.0", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "scaling threshold", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "progress", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.614 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_pg_recovery_event": { 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:44.615 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.616 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.619 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "max_completed_events": { 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_completed_events", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "50", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "sleep_interval": { 2024-03-28T18:49:44.621 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "sleep_interval", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "5", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "prometheus", 2024-03-28T18:49:44.622 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "cache": { 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "cache", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "exclude_perf_counters": { 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "exclude_perf_counters", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.625 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.626 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.627 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.628 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools": { 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.629 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "300", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.630 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "scrape_interval": { 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "scrape_interval", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15.0", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.631 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "::", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.632 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "9283", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "stale_cache_strategy": { 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stale_cache_strategy", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.633 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "log", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_behaviour": { 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_behaviour", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "default", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.634 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "default", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "error" 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "standby_error_status_code": { 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "standby_error_status_code", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "500", 2024-03-28T18:49:44.635 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "400", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "599", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_support", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.636 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.637 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.638 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.639 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.640 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_snap_create": { 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.641 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.642 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "trash_purge_schedule": { 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "trash_purge_schedule", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.643 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "restful", 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "enable_auth": { 2024-03-28T18:49:44.644 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enable_auth", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "key_file": { 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "key_file", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.645 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.646 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.647 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.648 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.649 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.650 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "server_addr": { 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_addr", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.651 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "server_port": { 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "server_port", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.652 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rgw", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.653 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.654 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.655 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.656 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rook", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.658 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.659 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.660 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.661 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "storage_class": { 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "storage_class", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.662 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "local", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "selftest", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.663 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.664 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.666 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.667 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "roption1": { 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption1", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "roption2": { 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "roption2", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "xyz", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption1": { 2024-03-28T18:49:44.669 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption1", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption2": { 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption2", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.670 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption3": { 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption3", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "float", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.671 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption4": { 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption4", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.672 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption5": { 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption5", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.673 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption6": { 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption6", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.674 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "rwoption7": { 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rwoption7", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "1", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "42", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.675 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "testkey": { 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testkey", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "testlkey": { 2024-03-28T18:49:44.676 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testlkey", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "testnewline": { 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "testnewline", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.677 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snap_schedule", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.678 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "allow_m_granularity": { 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "allow_m_granularity", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "dump_on_update": { 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "dump_on_update", 2024-03-28T18:49:44.679 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.680 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.682 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.683 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.684 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "stats", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.685 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.686 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.687 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "status", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.690 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.691 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.692 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.693 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telegraf", 2024-03-28T18:49:44.694 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "address": { 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "address", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.695 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "15", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.696 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.697 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.699 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.700 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "telemetry", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_basic": { 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_basic", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.701 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_crash": { 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_crash", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.702 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_device": { 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_device", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.703 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_ident": { 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_ident", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.704 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "channel_perf": { 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "channel_perf", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.705 INFO:teuthology.orchestra.run.smithi145.stdout: "contact": { 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "contact", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "description": { 2024-03-28T18:49:44.706 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "description", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "device_url": { 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "device_url", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "enabled": { 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "enabled", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "24", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "8", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.709 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "last_opt_revision": { 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "last_opt_revision", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "1", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.710 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard": { 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "leaderboard_description": { 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "leaderboard_description", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.712 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.713 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.715 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.716 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "organization": { 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "organization", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.717 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "proxy": { 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "proxy", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.718 INFO:teuthology.orchestra.run.smithi145.stdout: "url": { 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "url", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "test_orchestrator", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.720 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.721 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.722 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.723 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "volumes", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.724 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.728 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "max_concurrent_clones": { 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "max_concurrent_clones", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "4", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "periodic_async_work": { 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "periodic_async_work", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Periodically check for async work", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.730 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_delay": { 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "0", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "True", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "can_run": true, 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "error_string": "", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "module_options": { 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "discovery_interval": { 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "discovery_interval", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "uint", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "100", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.733 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "identifier": { 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "identifier", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.734 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "interval": { 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "interval", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "secs", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "60", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "log_level": { 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_level", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.736 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster": { 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.737 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_cluster_level": { 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_cluster_level", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "info", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [ 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "critical", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "debug", 2024-03-28T18:49:44.738 INFO:teuthology.orchestra.run.smithi145.stdout: "error", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "info", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "warning" 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "log_to_file": { 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "log_to_file", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "bool", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 1, 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "False", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.739 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_host": { 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_host", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.740 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_port": { 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_port", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "int", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "10051", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.741 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "zabbix_sender": { 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "zabbix_sender", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "str", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "level": "advanced", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "flags": 0, 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "min": "", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "max": "", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "enum_allowed": [], 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "desc": "", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "long_desc": "", 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "tags": [], 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: "see_also": [] 2024-03-28T18:49:44.742 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "services": { 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "dashboard": "https://172.21.15.145:8443/" 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "always_on_modules": { 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "octopus": [ 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "balancer", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "crash", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "devicehealth", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_autoscaler", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "progress", 2024-03-28T18:49:44.743 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_support", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "status", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "telemetry", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "volumes" 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "pacific": [ 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "balancer", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "crash", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "devicehealth", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_autoscaler", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "progress", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_support", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "status", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "telemetry", 2024-03-28T18:49:44.744 INFO:teuthology.orchestra.run.smithi145.stdout: "volumes" 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "quincy": [ 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "balancer", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "crash", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "devicehealth", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_autoscaler", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "progress", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_support", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "status", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "telemetry", 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "volumes" 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "reef": [ 2024-03-28T18:49:44.745 INFO:teuthology.orchestra.run.smithi145.stdout: "balancer", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "crash", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "devicehealth", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_autoscaler", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "progress", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_support", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "status", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "telemetry", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "volumes" 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "squid": [ 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "balancer", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "crash", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "devicehealth", 2024-03-28T18:49:44.746 INFO:teuthology.orchestra.run.smithi145.stdout: "orchestrator", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "pg_autoscaler", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "progress", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "rbd_support", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "status", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "telemetry", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "volumes" 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "last_failure_osd_epoch": 3, 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "active_clients": [ 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "devicehealth", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "addrvec": [ 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v2", 2024-03-28T18:49:44.747 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:0", 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 4218641686 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "libcephsqlite", 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "addrvec": [ 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v2", 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:0", 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 3894963170 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.748 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "rbd_support", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "addrvec": [ 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v2", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:0", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 3310240071 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "name": "volumes", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "addrvec": [ 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v2", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "172.21.15.145:0", 2024-03-28T18:49:44.749 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 3747630696 2024-03-28T18:49:44.750 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.750 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.750 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:44.750 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-03-28T18:49:44.750 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:49:44.751 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:43 smithi192 ceph-mon[36589]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 571 MiB/s wr, 674 op/s 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:43 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2976733029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:43 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1129053253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:43 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3316332837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:44 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1208348310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:43 smithi164 ceph-mon[36248]: pgmap v709: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 571 MiB/s wr, 674 op/s 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:43 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2976733029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:43 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1129053253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:43 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3316332837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:49:44.752 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:44 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1208348310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:49:44.752 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 msgr2=0x7fa0840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 0x7fa0840a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0980058c0 tx=0x7fa098005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 msgr2=0x7fa09c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 0x7fa09c197ec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9f0 tx=0x7fa08c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 shutdown_connections 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c1032a0 0x7fa09c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fa0840a2800 0x7fa0840a4cc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa09c1028e0 0x7fa09c197ec0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 --2- 172.21.15.145:0/1208348310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa09c1016f0 0x7fa09c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 >> 172.21.15.145:0/1208348310 conn(0x7fa09c0fcc40 msgr2=0x7fa09c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 shutdown_connections 2024-03-28T18:49:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:49:43.563+0000 7fa0a2d77640 1 -- 172.21.15.145:0/1208348310 wait complete. 2024-03-28T18:49:44.761 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T18:49:44.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T18:49:44.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:44 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1208348310' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:49:45.036 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:49:45.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:45 smithi145 ceph-mon[32151]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 551 MiB/s wr, 624 op/s 2024-03-28T18:49:45.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:45 smithi192 ceph-mon[36589]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 551 MiB/s wr, 624 op/s 2024-03-28T18:49:45.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:45 smithi164 ceph-mon[36248]: pgmap v710: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 551 MiB/s wr, 624 op/s 2024-03-28T18:49:47.474 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:47 smithi145 ceph-mon[32151]: pgmap v711: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 506 MiB/s wr, 510 op/s 2024-03-28T18:49:47.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:47 smithi192 ceph-mon[36589]: pgmap v711: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 506 MiB/s wr, 510 op/s 2024-03-28T18:49:47.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:47 smithi164 ceph-mon[36248]: pgmap v711: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 506 MiB/s wr, 510 op/s 2024-03-28T18:49:47.926 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:49:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:49:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 300, 2024-03-28T18:49:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 477, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 72878, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 87373, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 8, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 3, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.056687159, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.020563346, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.029975114, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.005666930, 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 475, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.097456933, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000205172 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 742, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000308136, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000415 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 288, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 279, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 804889, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 196664, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 10, 2024-03-28T18:49:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 5, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.065717892, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.024845202, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.022965962, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.006815364, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 278, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.029741356, 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000106983 2024-03-28T18:49:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 4, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000734, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000183 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 801808, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 142512 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:49:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 20, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 190, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 7265, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 385685, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 6, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 4, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.017180804, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.013173854, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.002491921, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.000229240, 2024-03-28T18:49:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 189, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.040251803, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000212972 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 378320 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "cct": { 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "total_workers": 1, 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "unhealthy_workers": 0 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-MDSRank": { 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 45, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.001044938, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000023220 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-PurgeQueue": { 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 2, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000006603, 2024-03-28T18:49:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000003301 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": { 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "request": 0, 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "reply": 0, 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "reply_latency": { 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_reply": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "forward": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_complete": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_keys": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_commit": 11, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_split": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_merge": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes": 10, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_top": 0, 2024-03-28T18:49:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_bottom": 10, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pin_tail": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pinned": 10, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_expired": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_with_caps": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "caps": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "subtrees": 1, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_hit": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_forward": 0, 2024-03-28T18:49:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_discover": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_remote_ino": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_lock": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "load_cent": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "q": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "exported": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "exported_inodes": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "imported": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "imported_inodes": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_dir_fetch": 0, 2024-03-28T18:49:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_peer_discover": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rfiles": 22, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rbytes": 10738150859, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rsnaps": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_set_tag": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:49:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_file_inodes": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_cap_release": 0, 2024-03-28T18:49:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "process_request_cap_release": 0 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_cache": { 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update_receipt": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_try_discover": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_send_discover": 1, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_handle_discover": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_delayed": 0, 2024-03-28T18:49:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_created": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_enqueued": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_reintegrated": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_migrated": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_processing": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_started": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_completed": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:49:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_exportdir": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_flush": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragstats": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_inodestats": 0 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_log": { 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "evlrg": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "evadd": 2, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "evex": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "evtrm": 0, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "ev": 2, 2024-03-28T18:49:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "evexg": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "evexd": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segadd": 2, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segex": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segtrm": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "seg": 2, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segmjr": 2, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segexg": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "segexd": 0, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "expos": 4194304, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "wrpos": 4194892, 2024-03-28T18:49:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "rdpos": 0, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "jlat": { 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 1, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.001947118, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.001947118 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "replayed": 0 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_mem": { 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "ino": 13, 2024-03-28T18:49:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "ino+": 13, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "ino-": 0, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dir": 11, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dir+": 12, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dir-": 1, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dn": 10, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dn+": 10, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "dn-": 0, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "cap": 0, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "cap+": 0, 2024-03-28T18:49:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "cap-": 0, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "rss": 51624, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "heap": 223612 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_server": { 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_client_request": 0, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_server_request": 0, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_request": 0, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_session": 0, 2024-03-28T18:49:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_peer_request": 0, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "req_create_latency": { 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getattr_latency": { 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getfilelock_latency": { 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "req_link_latency": { 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookup_latency": { 2024-03-28T18:49:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookuphash_latency": { 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupino_latency": { 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupname_latency": { 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupparent_latency": { 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupsnap_latency": { 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lssnap_latency": { 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mkdir_latency": { 2024-03-28T18:49:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mknod_latency": { 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mksnap_latency": { 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "req_open_latency": { 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "req_readdir_latency": { 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rename_latency": { 2024-03-28T18:49:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "req_renamesnap_latency": { 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "req_snapdiff_latency": { 2024-03-28T18:49:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmdir_latency": { 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmsnap_latency": { 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmxattr_latency": { 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setattr_latency": { 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setdirlayout_latency": { 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setfilelock_latency": { 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setlayout_latency": { 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setxattr_latency": { 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "req_symlink_latency": { 2024-03-28T18:49:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "req_unlink_latency": { 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:49:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getvxattr_latency": { 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_sessions": { 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "session_count": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "session_add": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "session_remove": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_open": 0, 2024-03-28T18:49:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_stale": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "total_load": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "average_load": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "avg_session_uptime": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mdthresh_evicted": 0 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "mempool": { 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_items": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:49:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_items": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:49:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:49:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_items": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_bytes": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_items": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_bytes": 238039, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_items": 75, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:49:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_items": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_bytes": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_items": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_items": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_items": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_bytes": 8984, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_items": 172, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_bytes": 0, 2024-03-28T18:49:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_items": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_bytes": 50032, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_items": 197, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_bytes": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_items": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_bytes": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_items": 0 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "objecter": { 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "op_active": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "op_laggy": 0, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send": 35, 2024-03-28T18:49:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send_bytes": 10710, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "op_resend": 33, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "op_reply": 35, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "op_latency": { 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 35, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.231994885, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.006628425 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "op_inflight": 0, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "oplen_avg": { 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 35, 2024-03-28T18:49:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 57 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "op": 35, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "op_r": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "op_w": 35, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "op_rmw": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "op_pg": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_stat": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_create": 10, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_read": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_write": 1, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writefull": 5, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writesame": 0, 2024-03-28T18:49:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_append": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_zero": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_truncate": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_delete": 6, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_mapext": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_sparse_read": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_clonerange": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_getxattr": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_setxattr": 20, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_rmxattr": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_call": 0, 2024-03-28T18:49:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_watch": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_notify": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_other": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_active": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_send": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_resend": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_ping": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_active": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_send": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_resend": 0, 2024-03-28T18:49:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_active": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_send": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_resend": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_active": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_send": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_resend": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "command_active": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "command_send": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "command_resend": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "map_epoch": 95, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "map_full": 0, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "map_inc": 4, 2024-03-28T18:49:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_sessions": 9, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_open": 9, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_close": 0, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_laggy": 0, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_wr": 14, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_rd": 0, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_del": 1 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "oft": { 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_objs": 1, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_updates": 0, 2024-03-28T18:49:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_removes": 0 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "osd-slow-ops": { 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_ops_count": 0 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "purge_queue": { 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_high_water": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed_ops": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed": 0, 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_item_in_journal": 0 2024-03-28T18:49:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 604, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 813714, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 604, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 604, 2024-03-28T18:49:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 813714, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_bytes": { 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 35, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 10710, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 35, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 10710, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_ops": { 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 1024, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 35, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 35, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 35, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 35, 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle-0x563181de6500": { 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 2, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 588, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 2, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 1, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 588, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.984 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:47.984 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:49:48.733 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-28T18:49:48.733 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-28T18:49:48.733 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T18:49:48.733 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T18:49:49.026 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:49:49.485 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:49 smithi145 ceph-mon[32151]: pgmap v712: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 373 MiB/s wr, 257 op/s 2024-03-28T18:49:49.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:49 smithi192 ceph-mon[36589]: pgmap v712: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 373 MiB/s wr, 257 op/s 2024-03-28T18:49:49.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:49 smithi164 ceph-mon[36248]: pgmap v712: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 373 MiB/s wr, 257 op/s 2024-03-28T18:49:51.596 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:51 smithi145 ceph-mon[32151]: pgmap v713: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 342 MiB/s wr, 232 op/s 2024-03-28T18:49:51.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:51 smithi192 ceph-mon[36589]: pgmap v713: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 342 MiB/s wr, 232 op/s 2024-03-28T18:49:51.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:51 smithi164 ceph-mon[36248]: pgmap v713: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 342 MiB/s wr, 232 op/s 2024-03-28T18:49:52.504 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:49:52.504 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:49:52.504 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 1003, 2024-03-28T18:49:52.504 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 998, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 171004362, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 382287, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 7, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 3, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.394594358, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.048623890, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.277059781, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.062570961, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 997, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.244351875, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000245087 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:52.505 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 772496, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 140912 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 489, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 666, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 210818696, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 428393, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 7, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 6, 2024-03-28T18:49:52.506 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.316358807, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.034991621, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.243680750, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.030111937, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 664, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.162958855, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000245419 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.507 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 273152 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 797, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 797, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 285476935, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 252629, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 6, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 5, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.458153205, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.022849795, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.363575957, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.062283030, 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:52.508 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 796, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.213911378, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000268732 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "cct": { 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "total_workers": 1, 2024-03-28T18:49:52.509 INFO:teuthology.orchestra.run.smithi145.stdout: "unhealthy_workers": 0 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-MDSRank": { 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 2965, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.114805883, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000038720 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-PurgeQueue": { 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:52.510 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": { 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "request": 0, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "reply": 0, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "reply_latency": { 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.511 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_reply": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "forward": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_complete": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_keys": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_commit": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_split": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_merge": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes": 28, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_top": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_bottom": 28, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pin_tail": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pinned": 28, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_expired": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_with_caps": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "caps": 0, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "subtrees": 2, 2024-03-28T18:49:52.512 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_hit": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_forward": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_discover": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_remote_ino": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_lock": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "load_cent": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "q": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "exported": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "exported_inodes": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "imported": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "imported_inodes": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_dir_fetch": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_peer_discover": 0, 2024-03-28T18:49:52.513 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rfiles": 22, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rbytes": 10738150859, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rsnaps": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_set_tag": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_file_inodes": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:49:52.514 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_cap_release": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "process_request_cap_release": 0 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_cache": { 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update_receipt": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_try_discover": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_send_discover": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_handle_discover": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_delayed": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:49:52.515 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_created": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_enqueued": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_reintegrated": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_migrated": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_processing": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_started": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_completed": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_exportdir": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_flush": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragstats": 0, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_inodestats": 0 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.516 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_log": { 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evlrg": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evadd": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evex": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evtrm": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "ev": 334662, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evexg": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "evexd": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segadd": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segex": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segtrm": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "seg": 9, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segmjr": 44, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segexg": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "segexd": 0, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "expos": 832796576, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "wrpos": 849454840, 2024-03-28T18:49:52.517 INFO:teuthology.orchestra.run.smithi145.stdout: "rdpos": 849452819, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "jlat": { 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "replayed": 334662 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_mem": { 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "ino": 16, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "ino+": 30, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "ino-": 14, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "dir": 7, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "dir+": 11, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "dir-": 4, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "dn": 28, 2024-03-28T18:49:52.518 INFO:teuthology.orchestra.run.smithi145.stdout: "dn+": 49, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "dn-": 21, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "cap": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "cap+": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "cap-": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "rss": 55636, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "heap": 215420 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_server": { 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_client_request": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_server_request": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_request": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_session": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_peer_request": 0, 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "req_create_latency": { 2024-03-28T18:49:52.519 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getattr_latency": { 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getfilelock_latency": { 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "req_link_latency": { 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.520 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookup_latency": { 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookuphash_latency": { 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupino_latency": { 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupname_latency": { 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupparent_latency": { 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupsnap_latency": { 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lssnap_latency": { 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mkdir_latency": { 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.523 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mknod_latency": { 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mksnap_latency": { 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "req_open_latency": { 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.524 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "req_readdir_latency": { 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rename_latency": { 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "req_renamesnap_latency": { 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "req_snapdiff_latency": { 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmdir_latency": { 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmsnap_latency": { 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmxattr_latency": { 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setattr_latency": { 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setdirlayout_latency": { 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setfilelock_latency": { 2024-03-28T18:49:52.527 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setlayout_latency": { 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setxattr_latency": { 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.528 INFO:teuthology.orchestra.run.smithi145.stdout: "req_symlink_latency": { 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "req_unlink_latency": { 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getvxattr_latency": { 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.529 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_sessions": { 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "session_count": 1, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "session_add": 1, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "session_remove": 0, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_open": 0, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_stale": 0, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "total_load": 0, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "average_load": 0, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "avg_session_uptime": 1811, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "mdthresh_evicted": 0 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "mempool": { 2024-03-28T18:49:52.530 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_items": 0, 2024-03-28T18:49:52.531 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_bytes": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_items": 0, 2024-03-28T18:49:52.532 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_bytes": 238964, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_items": 84, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_items": 0, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_bytes": 8984, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_items": 172, 2024-03-28T18:49:52.533 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_bytes": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_items": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_bytes": 70440, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_items": 1276, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_bytes": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_items": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_bytes": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_items": 0 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "objecter": { 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "op_active": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "op_laggy": 0, 2024-03-28T18:49:52.534 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send": 2040, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send_bytes": 26, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op_resend": 4, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op_reply": 2040, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op_latency": { 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 2040, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 55.100815830, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.027010203 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op_inflight": 0, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "oplen_avg": { 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 2040, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 2041 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.535 INFO:teuthology.orchestra.run.smithi145.stdout: "op": 2040, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "op_r": 2040, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "op_w": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "op_rmw": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "op_pg": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_stat": 1238, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_create": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_read": 799, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_write": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writefull": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writesame": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_append": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_zero": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_truncate": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_delete": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_mapext": 0, 2024-03-28T18:49:52.536 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_sparse_read": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_clonerange": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_getxattr": 2, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_setxattr": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_rmxattr": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_call": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_watch": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_notify": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_other": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_active": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_send": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_resend": 0, 2024-03-28T18:49:52.537 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_ping": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_active": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_send": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_resend": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_active": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_send": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_resend": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_active": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_send": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_resend": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "command_active": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "command_send": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "command_resend": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "map_epoch": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "map_full": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "map_inc": 0, 2024-03-28T18:49:52.538 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_sessions": 11, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_open": 11, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_close": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_laggy": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_wr": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_rd": 2, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_del": 0 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "oft": { 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_objs": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_updates": 0, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_removes": 0 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "osd-slow-ops": { 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_ops_count": 0 2024-03-28T18:49:52.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "purge_queue": { 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_high_water": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed_ops": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_item_in_journal": 0 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 2285, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 667078785, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:52.540 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 2285, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 2285, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 667078785, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_bytes": { 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:52.541 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 2040, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 665969385, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 2040, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 665969385, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_ops": { 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:52.542 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 1024, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 2040, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 2040, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 2040, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 2040, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle-0x55c4559e46e0": { 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:52.545 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:52.546 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.546 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:52.546 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:49:53.457 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-03-28T18:49:53.457 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-03-28T18:49:53.457 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T18:49:53.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T18:49:53.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:53 smithi192 ceph-mon[36589]: pgmap v714: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 167 MiB/s wr, 162 op/s 2024-03-28T18:49:53.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:53 smithi164 ceph-mon[36248]: pgmap v714: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 167 MiB/s wr, 162 op/s 2024-03-28T18:49:53.715 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:49:53.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:53 smithi145 ceph-mon[32151]: pgmap v714: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 167 MiB/s wr, 162 op/s 2024-03-28T18:49:55.524 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:55 smithi145 ceph-mon[32151]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 116 MiB/s wr, 223 op/s 2024-03-28T18:49:55.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:55 smithi192 ceph-mon[36589]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 116 MiB/s wr, 223 op/s 2024-03-28T18:49:55.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:55 smithi164 ceph-mon[36248]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 116 MiB/s wr, 223 op/s 2024-03-28T18:49:55.798 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 513, 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 515, 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 172313, 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 163135, 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 6, 2024-03-28T18:49:55.799 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 2, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.094471340, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.024746369, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.019543751, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.044748688, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 513, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.056389315, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000109920 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.800 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 764, 2024-03-28T18:49:55.801 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 755, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 924424, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 303541, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 4, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 3, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.121935532, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.029958763, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.035140109, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.055439938, 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:55.802 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 755, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.095852135, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000126956 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.803 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 789136, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 140032 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 4, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 178, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 1508, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 382389, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 5, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 4, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.017124019, 2024-03-28T18:49:55.804 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.013472351, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.001698336, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.000062922, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 176, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.009438115, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000053625 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.805 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 381424 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "cct": { 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "total_workers": 1, 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "unhealthy_workers": 0 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.806 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-MDSRank": { 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 2053, 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.068976144, 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000033597 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "finisher-PurgeQueue": { 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "queue_len": 0, 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "complete_latency": { 2024-03-28T18:49:55.807 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "mds": { 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "request": 0, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "reply": 0, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "reply_latency": { 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.808 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_reply": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "forward": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_complete": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_fetch_keys": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_commit": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_split": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_merge": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes": 0, 2024-03-28T18:49:55.809 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_top": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_bottom": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pin_tail": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_pinned": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_expired": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "inodes_with_caps": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "caps": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "subtrees": 1, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_hit": 0, 2024-03-28T18:49:55.810 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_forward": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_discover": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_remote_ino": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "traverse_lock": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "load_cent": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "q": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "exported": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "exported_inodes": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "imported": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "imported_inodes": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_dir_fetch": 0, 2024-03-28T18:49:55.811 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "openino_peer_discover": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rfiles": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rbytes": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "root_rsnaps": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_set_tag": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:49:55.812 INFO:teuthology.orchestra.run.smithi145.stdout: "scrub_file_inodes": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_cap_release": 0, 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: "process_request_cap_release": 0 2024-03-28T18:49:55.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_cache": { 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_update_receipt": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_try_discover": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_send_discover": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "dir_handle_discover": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_delayed": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:49:55.814 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_created": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_enqueued": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_reintegrated": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "strays_migrated": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_processing": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_started": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "recovery_completed": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_exportdir": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_flush": 0, 2024-03-28T18:49:55.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_fragstats": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "ireq_inodestats": 0 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_log": { 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evlrg": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evadd": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evex": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evtrm": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "ev": 2, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evexg": 0, 2024-03-28T18:49:55.816 INFO:teuthology.orchestra.run.smithi145.stdout: "evexd": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segadd": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segex": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segtrm": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "seg": 2, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segmjr": 2, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segexg": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "segexd": 0, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "expos": 4194304, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "wrpos": 4194892, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "rdpos": 4194352, 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "jlat": { 2024-03-28T18:49:55.817 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "replayed": 2 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_mem": { 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "ino": 3, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "ino+": 3, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "ino-": 0, 2024-03-28T18:49:55.818 INFO:teuthology.orchestra.run.smithi145.stdout: "dir": 1, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "dir+": 1, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "dir-": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "dn": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "dn+": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "dn-": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "cap": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "cap+": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "cap-": 0, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "rss": 50116, 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: "heap": 215420 2024-03-28T18:49:55.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_server": { 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_client_request": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "dispatch_server_request": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_request": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_client_session": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "handle_peer_request": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "req_create_latency": { 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getattr_latency": { 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getfilelock_latency": { 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "req_link_latency": { 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookup_latency": { 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.822 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookuphash_latency": { 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupino_latency": { 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.823 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupname_latency": { 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupparent_latency": { 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.824 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lookupsnap_latency": { 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "req_lssnap_latency": { 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mkdir_latency": { 2024-03-28T18:49:55.825 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.826 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.826 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.826 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mknod_latency": { 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "req_mksnap_latency": { 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.827 INFO:teuthology.orchestra.run.smithi145.stdout: "req_open_latency": { 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "req_readdir_latency": { 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rename_latency": { 2024-03-28T18:49:55.828 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "req_renamesnap_latency": { 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "req_snapdiff_latency": { 2024-03-28T18:49:55.829 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmdir_latency": { 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmsnap_latency": { 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.830 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "req_rmxattr_latency": { 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setattr_latency": { 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.831 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setdirlayout_latency": { 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setfilelock_latency": { 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setlayout_latency": { 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.832 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "req_setxattr_latency": { 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "req_symlink_latency": { 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.833 INFO:teuthology.orchestra.run.smithi145.stdout: "req_unlink_latency": { 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "req_getvxattr_latency": { 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.834 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_sessions": { 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "session_count": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "session_add": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "session_remove": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_open": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "sessions_stale": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "total_load": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "average_load": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "avg_session_uptime": 0, 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: "mdthresh_evicted": 0 2024-03-28T18:49:55.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "mempool": { 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_items": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:49:55.836 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:49:55.837 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:49:55.838 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_bytes": 197543, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_items": 56, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_items": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_bytes": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_items": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_items": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_items": 0, 2024-03-28T18:49:55.839 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_bytes": 8984, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_items": 172, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_bytes": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_items": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_bytes": 8320, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_items": 13, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_bytes": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_items": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_bytes": 0, 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_items": 0 2024-03-28T18:49:55.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "objecter": { 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_active": 0, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_laggy": 0, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send": 1030, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_send_bytes": 21, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_resend": 3, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_reply": 1030, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "op_latency": { 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 1030, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 311.124284049, 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.302062411 2024-03-28T18:49:55.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op_inflight": 0, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "oplen_avg": { 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 1030, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 1031 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op": 1030, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op_r": 1030, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op_w": 0, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op_rmw": 0, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "op_pg": 0, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_stat": 512, 2024-03-28T18:49:55.842 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_create": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_read": 516, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_write": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writefull": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_writesame": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_append": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_zero": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_truncate": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_delete": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_mapext": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_sparse_read": 0, 2024-03-28T18:49:55.843 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_clonerange": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_getxattr": 1, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_setxattr": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_rmxattr": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_call": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_watch": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_notify": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls": 0, 2024-03-28T18:49:55.844 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "osdop_other": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_active": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_send": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_resend": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "linger_ping": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_active": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_send": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolop_resend": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_active": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_send": 0, 2024-03-28T18:49:55.845 INFO:teuthology.orchestra.run.smithi145.stdout: "poolstat_resend": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_active": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_send": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "statfs_resend": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "command_active": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "command_send": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "command_resend": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "map_epoch": 95, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "map_full": 0, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "map_inc": 4, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_sessions": 6, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_open": 6, 2024-03-28T18:49:55.846 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_session_close": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_laggy": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_wr": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_rd": 2, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_del": 0 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "oft": { 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_objs": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_updates": 0, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "omap_total_removes": 0 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "osd-slow-ops": { 2024-03-28T18:49:55.847 INFO:teuthology.orchestra.run.smithi145.stdout: "slow_ops_count": 0 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "purge_queue": { 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executing_high_water": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed_ops": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_executed": 0, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "pq_item_in_journal": 0 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:49:55.848 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 1277, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 967673, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 1277, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 1277, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 967673, 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:55.849 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_bytes": { 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:55.850 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 1030, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 593, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 1030, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 593, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-objecter_ops": { 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 1024, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 1030, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 1030, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 1030, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 1030, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 0, 2024-03-28T18:49:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 0, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-write_buf_throttle-0x56176fcc06e0": { 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 3758096384, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 0, 2024-03-28T18:49:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:55.855 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:49:57.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:57 smithi192 ceph-mon[36589]: pgmap v716: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 95 MiB/s wr, 383 op/s 2024-03-28T18:49:57.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:57 smithi164 ceph-mon[36248]: pgmap v716: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 95 MiB/s wr, 383 op/s 2024-03-28T18:49:57.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:57 smithi145 ceph-mon[32151]: pgmap v716: 129 pgs: 129 active+clean; 10 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 95 MiB/s wr, 383 op/s 2024-03-28T18:49:57.937 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-28T18:49:57.938 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-28T18:49:57.938 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T18:49:57.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T18:49:58.193 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:49:58.605 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:49:58 smithi145 ceph-mon[32151]: pgmap v717: 129 pgs: 129 active+clean; 9.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 67 MiB/s wr, 394 op/s 2024-03-28T18:49:58.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:49:58 smithi192 ceph-mon[36589]: pgmap v717: 129 pgs: 129 active+clean; 9.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 67 MiB/s wr, 394 op/s 2024-03-28T18:49:58.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:49:58 smithi164 ceph-mon[36248]: pgmap v717: 129 pgs: 129 active+clean; 9.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 67 MiB/s wr, 394 op/s 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 0, 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 2, 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 0, 2024-03-28T18:49:59.925 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 0, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 4, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 0, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.001653769, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.000397087, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:59.926 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 1, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 180, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 160, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 109200, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 3, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 2, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.019269265, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.011409718, 2024-03-28T18:49:59.927 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.001908704, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 178, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.020514780, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000115251 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 109200 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:49:59.928 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_messages": 245, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages": 238, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_bytes": 759968, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_bytes": 142240, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_created_connections": 2, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_active_connections": 1, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_total_time": 0.042190408, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_send_time": 0.017106122, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_recv_time": 0.019239421, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_running_fast_dispatch_time": 0.006712971, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 238, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.028258574, 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000118733 2024-03-28T18:49:59.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_recv_encrypted_bytes": 759968, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "msgr_send_encrypted_bytes": 142240 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "cct": { 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "total_workers": 0, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "unhealthy_workers": 0 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "mempool": { 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "bloom_filter_items": 0, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:49:59.930 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_extent_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_blob_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:49:59.931 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_txc_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluestore_writing_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:49:59.932 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_bytes": 149024, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_anon_items": 31, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "buffer_meta_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_mapbl_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_pglog_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_bytes": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "pgmap_items": 0, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_bytes": 928, 2024-03-28T18:49:59.933 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_co_items": 2, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_bytes": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_1_items": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_bytes": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "unittest_2_items": 0 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "val": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "max": 104857600, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "get_started": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "get": 242, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "get_sum": 722912, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_fail": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "get_or_fail_success": 242, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "take": 0, 2024-03-28T18:49:59.934 INFO:teuthology.orchestra.run.smithi145.stdout: "take_sum": 0, 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "put": 242, 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "put_sum": 722912, 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "wait": { 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgcount": 0, 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "sum": 0.000000000, 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: "avgtime": 0.000000000 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-03-28T18:49:59.935 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-03-28T18:50:00.202 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:00 smithi145 ceph-mon[32151]: overall HEALTH_OK 2024-03-28T18:50:00.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:00 smithi192 ceph-mon[36589]: overall HEALTH_OK 2024-03-28T18:50:00.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:00 smithi164 ceph-mon[36248]: overall HEALTH_OK 2024-03-28T18:50:00.668 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T18:50:00.668 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T18:50:00.668 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T18:50:00.668 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T18:50:01.178 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:50:01.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:01 smithi164 ceph-mon[36248]: pgmap v718: 129 pgs: 129 active+clean; 8.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 551 op/s 2024-03-28T18:50:01.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:01 smithi192 ceph-mon[36589]: pgmap v718: 129 pgs: 129 active+clean; 8.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 551 op/s 2024-03-28T18:50:01.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:01 smithi145 ceph-mon[32151]: pgmap v718: 129 pgs: 129 active+clean; 8.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 74 MiB/s wr, 551 op/s 2024-03-28T18:50:03.351 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:03 smithi145 ceph-mon[32151]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 60 MiB/s wr, 581 op/s 2024-03-28T18:50:03.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:03 smithi192 ceph-mon[36589]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 60 MiB/s wr, 581 op/s 2024-03-28T18:50:03.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:03 smithi164 ceph-mon[36248]: pgmap v719: 129 pgs: 129 active+clean; 8.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 60 MiB/s wr, 581 op/s 2024-03-28T18:50:03.534 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-28T18:50:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 598, 2024-03-28T18:50:03.534 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 774, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 899510, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 371673, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 7, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.085644669, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.032559895, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.042717047, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.010858334, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 774, 2024-03-28T18:50:03.537 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.074179115, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000095838 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 777, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000301476, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000388 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 822256, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 144656 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 19, 2024-03-28T18:50:03.538 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 193, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 4456, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 434744, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 11, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 6, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.013459207, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.007176146, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.003639373, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.001243542, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 191, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.018462955, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000096664 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.539 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 11, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000001595, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000145 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 385360 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 26, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 28, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 5716, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 14227, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 7, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 5, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.009983691, 2024-03-28T18:50:03.540 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.002000978, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.006929312, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.000371600, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 26, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.021946803, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000844107 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 21, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000005654, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000269 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:50:03.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "cct": { 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "total_workers": 1, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "unhealthy_workers": 0 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-MDSRank": { 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 43, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.001721202, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000040027 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:03.542 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000008108, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000004054 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": { 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "request": 1, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "reply": 1, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "reply_latency": { 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 1, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000087008, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000087008 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_reply": 0, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "forward": 0, 2024-03-28T18:50:03.543 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_commit": 11, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_split": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_merge": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes": 10, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_top": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_bottom": 10, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pinned": 10, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_expired": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_with_caps": 1, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": 1, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "subtrees": 1, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_hit": 0, 2024-03-28T18:50:03.544 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_forward": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_discover": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_lock": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "load_cent": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "q": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "exported": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "exported_inodes": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "imported": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "imported_inodes": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rfiles": 22, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rbytes": 10738150859, 2024-03-28T18:50:03.545 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rsnaps": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:03.546 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "process_request_cap_release": 0 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_cache": { 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_try_discover": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_send_discover": 1, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:03.547 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_created": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_enqueued": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_migrated": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_started": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_completed": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_flush": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_inodestats": 0 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.548 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_log": { 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evlrg": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evadd": 5, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evex": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evtrm": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "ev": 5, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evexg": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "evexd": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segadd": 2, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segex": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segtrm": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "seg": 2, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segmjr": 2, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segexg": 0, 2024-03-28T18:50:03.549 INFO:teuthology.orchestra.run.smithi164.stdout: "segexd": 0, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "expos": 4194304, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "wrpos": 4195732, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "rdpos": 0, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "jlat": { 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 4, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.006347405, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.001586851 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "replayed": 0 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_mem": { 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "ino": 13, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "ino+": 13, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "ino-": 0, 2024-03-28T18:50:03.550 INFO:teuthology.orchestra.run.smithi164.stdout: "dir": 11, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dir+": 12, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dir-": 1, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dn": 10, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dn+": 10, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dn-": 0, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "cap": 1, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "cap+": 1, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "cap-": 0, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "rss": 48404, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "heap": 223612 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_server": { 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_client_request": 1, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_request": 1, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_session": 14, 2024-03-28T18:50:03.551 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_peer_request": 0, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "req_create_latency": { 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getattr_latency": { 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.552 INFO:teuthology.orchestra.run.smithi164.stdout: "req_link_latency": { 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookup_latency": { 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupino_latency": { 2024-03-28T18:50:03.553 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 1, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000087008, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000087008 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupname_latency": { 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.554 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lssnap_latency": { 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mkdir_latency": { 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mknod_latency": { 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mksnap_latency": { 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "req_open_latency": { 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "req_readdir_latency": { 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rename_latency": { 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.557 INFO:teuthology.orchestra.run.smithi164.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmdir_latency": { 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.558 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setattr_latency": { 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:03.559 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setlayout_latency": { 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.560 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setxattr_latency": { 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "req_symlink_latency": { 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "req_unlink_latency": { 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:03.561 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_sessions": { 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "session_count": 1, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "session_add": 2, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "session_remove": 1, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_open": 1, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_stale": 0, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "total_load": 1, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "average_load": 0, 2024-03-28T18:50:03.562 INFO:teuthology.orchestra.run.smithi164.stdout: "avg_session_uptime": 707, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "mempool": { 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:03.563 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:03.564 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_bytes": 238487, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_items": 79, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:03.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_items": 172, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_items": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_bytes": 50408, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_items": 200, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_items": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_items": 0 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "objecter": { 2024-03-28T18:50:03.566 INFO:teuthology.orchestra.run.smithi164.stdout: "op_active": 0, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_laggy": 0, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send": 40, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send_bytes": 11730, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_resend": 28, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_reply": 40, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_latency": { 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 40, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.219996303, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.005499907 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op_inflight": 0, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "oplen_avg": { 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 40, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 62 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.567 INFO:teuthology.orchestra.run.smithi164.stdout: "op": 40, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "op_r": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "op_w": 40, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "op_rmw": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "op_pg": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_stat": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_create": 10, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_read": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_write": 4, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writefull": 7, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writesame": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_append": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_zero": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_truncate": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_delete": 6, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_mapext": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:03.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_getxattr": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_setxattr": 20, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_call": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_watch": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_notify": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_other": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_active": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_send": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_resend": 0, 2024-03-28T18:50:03.569 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_ping": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_active": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_send": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_resend": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_active": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_send": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_resend": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_active": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_send": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_resend": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "command_active": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "command_send": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "command_resend": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "map_epoch": 95, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "map_full": 0, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "map_inc": 4, 2024-03-28T18:50:03.570 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_sessions": 9, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_open": 9, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_close": 0, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_laggy": 0, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_wr": 14, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_rd": 0, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_del": 1 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "oft": { 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_objs": 1, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_updates": 0, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_removes": 0 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "osd-slow-ops": { 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_ops_count": 0 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.571 INFO:teuthology.orchestra.run.smithi164.stdout: "purge_queue": { 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 639, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 834772, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 639, 2024-03-28T18:50:03.572 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 639, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 834772, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:03.573 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 40, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 11730, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 40, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 11730, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 1024, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:03.574 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 40, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 40, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 40, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 40, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:03.575 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle-0x55fb5e444500": { 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:03.576 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 5, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 1428, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 5, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 4, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 1428, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:03.577 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-28T18:50:04.875 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T18:50:04.875 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T18:50:04.875 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T18:50:04.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T18:50:05.194 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:50:05.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:05 smithi164 ceph-mon[36248]: pgmap v720: 129 pgs: 129 active+clean; 7.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 61 MiB/s wr, 576 op/s 2024-03-28T18:50:05.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:05 smithi192 ceph-mon[36589]: pgmap v720: 129 pgs: 129 active+clean; 7.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 61 MiB/s wr, 576 op/s 2024-03-28T18:50:05.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:05 smithi145 ceph-mon[32151]: pgmap v720: 129 pgs: 129 active+clean; 7.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 61 MiB/s wr, 576 op/s 2024-03-28T18:50:07.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:07 smithi164 ceph-mon[36248]: pgmap v721: 129 pgs: 129 active+clean; 6.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 64 MiB/s wr, 520 op/s 2024-03-28T18:50:07.328 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:07 smithi145 ceph-mon[32151]: pgmap v721: 129 pgs: 129 active+clean; 6.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 64 MiB/s wr, 520 op/s 2024-03-28T18:50:07.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:07 smithi192 ceph-mon[36589]: pgmap v721: 129 pgs: 129 active+clean; 6.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 64 MiB/s wr, 520 op/s 2024-03-28T18:50:07.713 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 1083, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 1073, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 1082944, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 406632, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 5, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 1, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.125605233, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.041730670, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.042619852, 2024-03-28T18:50:07.714 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.036443324, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 1072, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.057993180, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000054098 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 804736, 2024-03-28T18:50:07.715 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 143328 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 3, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 182, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 716, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 386202, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 4, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.009743831, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.007157038, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.001234825, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.000059283, 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:07.716 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 180, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.009308486, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000051713 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 385504 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:07.717 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 1547, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 1547, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 397923, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 491129, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 4, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.104244640, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.023336105, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.028017692, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.047523791, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 1546, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.061671680, 2024-03-28T18:50:07.718 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000039891 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "cct": { 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "total_workers": 1, 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: "unhealthy_workers": 0 2024-03-28T18:50:07.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-MDSRank": { 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 3927, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.106083570, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000027013 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.720 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": { 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "request": 0, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "reply": 0, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "reply_latency": { 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_reply": 0, 2024-03-28T18:50:07.721 INFO:teuthology.orchestra.run.smithi164.stdout: "forward": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_commit": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_split": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_merge": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_top": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_bottom": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pinned": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_expired": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_with_caps": 0, 2024-03-28T18:50:07.722 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "subtrees": 1, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_hit": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_forward": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_discover": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_lock": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "load_cent": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "q": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "exported": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "exported_inodes": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "imported": 0, 2024-03-28T18:50:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: "imported_inodes": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rfiles": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rbytes": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rsnaps": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "process_request_cap_release": 0 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_cache": { 2024-03-28T18:50:07.725 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_try_discover": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_send_discover": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_created": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_enqueued": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_migrated": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:07.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_started": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_completed": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_flush": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_inodestats": 0 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_log": { 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "evlrg": 0, 2024-03-28T18:50:07.727 INFO:teuthology.orchestra.run.smithi164.stdout: "evadd": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "evex": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "evtrm": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "ev": 5, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "evexg": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "evexd": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segadd": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segex": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segtrm": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "seg": 2, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segmjr": 2, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segexg": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "segexd": 0, 2024-03-28T18:50:07.728 INFO:teuthology.orchestra.run.smithi164.stdout: "expos": 4194304, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "wrpos": 4195732, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "rdpos": 4195591, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "jlat": { 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "replayed": 5 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_mem": { 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ino": 3, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ino+": 3, 2024-03-28T18:50:07.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ino-": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dir": 1, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dir+": 1, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dir-": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dn": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dn+": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "dn-": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "cap": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "cap+": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "cap-": 0, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "rss": 46036, 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: "heap": 215420 2024-03-28T18:50:07.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_server": { 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_client_request": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_request": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_session": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_peer_request": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "req_create_latency": { 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getattr_latency": { 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.731 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "req_link_latency": { 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.732 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookup_latency": { 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupino_latency": { 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.733 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupname_latency": { 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:07.734 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lssnap_latency": { 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mkdir_latency": { 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mknod_latency": { 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.736 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mksnap_latency": { 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "req_open_latency": { 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "req_readdir_latency": { 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.737 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rename_latency": { 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.738 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmdir_latency": { 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:07.739 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setattr_latency": { 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setlayout_latency": { 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setxattr_latency": { 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "req_symlink_latency": { 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "req_unlink_latency": { 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:07.742 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_sessions": { 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "session_count": 1, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "session_add": 2, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "session_remove": 1, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_open": 2, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_stale": 0, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "total_load": 0, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "average_load": 0, 2024-03-28T18:50:07.743 INFO:teuthology.orchestra.run.smithi164.stdout: "avg_session_uptime": 3360, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "mempool": { 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:07.744 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:07.745 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_bytes": 181320, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_items": 47, 2024-03-28T18:50:07.746 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_items": 172, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_items": 0, 2024-03-28T18:50:07.747 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_bytes": 8320, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_items": 13, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_items": 0, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_items": 0 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "objecter": { 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_active": 0, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_laggy": 0, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send": 2373, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send_bytes": 21, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_resend": 3, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_reply": 2373, 2024-03-28T18:50:07.748 INFO:teuthology.orchestra.run.smithi164.stdout: "op_latency": { 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2373, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 14.754757728, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.006217765 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op_inflight": 0, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "oplen_avg": { 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2373, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 2374 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op": 2373, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op_r": 2373, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op_w": 0, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op_rmw": 0, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "op_pg": 0, 2024-03-28T18:50:07.749 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_stat": 1536, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_create": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_read": 835, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_write": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writefull": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writesame": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_append": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_zero": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_truncate": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_delete": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_mapext": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:07.750 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_getxattr": 1, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_setxattr": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_call": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_watch": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_notify": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_other": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_active": 0, 2024-03-28T18:50:07.751 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_send": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_resend": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_ping": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_active": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_send": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_resend": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_active": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_send": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_resend": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_active": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_send": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_resend": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "command_active": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "command_send": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "command_resend": 0, 2024-03-28T18:50:07.752 INFO:teuthology.orchestra.run.smithi164.stdout: "map_epoch": 95, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "map_full": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "map_inc": 4, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_sessions": 4, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_open": 4, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_close": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_laggy": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_wr": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_rd": 2, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_del": 0 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "oft": { 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_objs": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_updates": 0, 2024-03-28T18:50:07.753 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_removes": 0 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "osd-slow-ops": { 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_ops_count": 0 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "purge_queue": { 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed": 0, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.754 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 2629, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 1228878, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 2629, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2629, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 1228878, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.755 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 2373, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 1433, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2373, 2024-03-28T18:50:07.756 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 1433, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 1024, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:07.757 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 2373, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 2373, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2373, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 2373, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:07.758 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle-0x564f5fe486e0": { 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:07.760 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:07.761 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:07.761 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:07.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:07.761 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-28T18:50:08.559 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T18:50:08.560 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T18:50:08.560 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T18:50:08.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T18:50:08.825 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:50:09.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:09 smithi164 ceph-mon[36248]: pgmap v722: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 47 MiB/s wr, 331 op/s 2024-03-28T18:50:09.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:09 smithi192 ceph-mon[36589]: pgmap v722: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 47 MiB/s wr, 331 op/s 2024-03-28T18:50:09.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:09 smithi145 ceph-mon[32151]: pgmap v722: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 47 MiB/s wr, 331 op/s 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 37, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 37, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 10090, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 55046, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 13, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:11.017 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.010680565, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.002018660, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.006851642, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.000296663, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 36, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.012712356, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000353121 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 25, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000004995, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000199 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:50:11.018 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 298, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 287, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 815873, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 153801, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 6, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 5, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.039987418, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.013865557, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.022625537, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.006655662, 2024-03-28T18:50:11.019 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 286, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.033808799, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000118212 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 812336, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 147504 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 306, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 671, 2024-03-28T18:50:11.020 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 78588, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 603925, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 6, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.051357478, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.022523359, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.021962626, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.004377914, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 669, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.060689671, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000090716 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 774, 2024-03-28T18:50:11.021 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000308715, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000398 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 386928 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "cct": { 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "total_workers": 1, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "unhealthy_workers": 0 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-MDSRank": { 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 47, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.001129537, 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000024032 2024-03-28T18:50:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000004280, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000002140 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": { 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "request": 1, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "reply": 0, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "reply_latency": { 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_reply": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "forward": 1, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_commit": 11, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_split": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_merge": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes": 10, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_top": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_bottom": 10, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pinned": 10, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_expired": 0, 2024-03-28T18:50:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_with_caps": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "subtrees": 1, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_hit": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_forward": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_discover": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_lock": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "load_cent": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "q": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "exported": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "exported_inodes": 0, 2024-03-28T18:50:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "imported": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "imported_inodes": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_peer_discover": 1, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rfiles": 22, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rbytes": 10738150859, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rsnaps": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "process_request_cap_release": 0 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_cache": { 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_try_discover": 0, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_send_discover": 1, 2024-03-28T18:50:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_created": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_enqueued": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_migrated": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_started": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_completed": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_flush": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_inodestats": 0 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_log": { 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evlrg": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evadd": 3, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evex": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evtrm": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "ev": 3, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evexg": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "evexd": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "segadd": 2, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "segex": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "segtrm": 0, 2024-03-28T18:50:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "seg": 2, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "segmjr": 2, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "segexg": 0, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "segexd": 0, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "expos": 4194304, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "wrpos": 4195200, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "rdpos": 0, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "jlat": { 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.003130327, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.001565163 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "replayed": 0 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_mem": { 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "ino": 13, 2024-03-28T18:50:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "ino+": 13, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "ino-": 0, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dir": 11, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dir+": 12, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dir-": 1, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dn": 10, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dn+": 10, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dn-": 0, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "cap": 0, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "cap+": 0, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "cap-": 0, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "rss": 48792, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "heap": 223612 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_server": { 2024-03-28T18:50:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_client_request": 1, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_request": 1, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_session": 10, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_peer_request": 0, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "req_create_latency": { 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getattr_latency": { 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "req_link_latency": { 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookup_latency": { 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupino_latency": { 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupname_latency": { 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lssnap_latency": { 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mkdir_latency": { 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mknod_latency": { 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mksnap_latency": { 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "req_open_latency": { 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.036 INFO:teuthology.orchestra.run.smithi164.stdout: "req_readdir_latency": { 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rename_latency": { 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.037 INFO:teuthology.orchestra.run.smithi164.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmdir_latency": { 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.038 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setattr_latency": { 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.039 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setlayout_latency": { 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setxattr_latency": { 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "req_symlink_latency": { 2024-03-28T18:50:11.040 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "req_unlink_latency": { 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_sessions": { 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "session_count": 1, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "session_add": 1, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "session_remove": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_open": 1, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_stale": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "total_load": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "average_load": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "avg_session_uptime": 715, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "mempool": { 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:11.042 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:11.043 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:11.044 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_bytes": 238343, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_items": 79, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_items": 172, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_items": 0, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_bytes": 50040, 2024-03-28T18:50:11.045 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_items": 198, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_items": 0, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_items": 0 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "objecter": { 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_active": 0, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_laggy": 0, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send": 37, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send_bytes": 11108, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_resend": 33, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_reply": 37, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "op_latency": { 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 37, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.172997089, 2024-03-28T18:50:11.046 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.004675597 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op_inflight": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "oplen_avg": { 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 37, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 59 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op": 37, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op_r": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op_w": 37, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op_rmw": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "op_pg": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_stat": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_create": 10, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_read": 0, 2024-03-28T18:50:11.047 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_write": 2, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writefull": 6, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writesame": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_append": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_zero": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_truncate": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_delete": 6, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_mapext": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_getxattr": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_setxattr": 20, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_call": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_watch": 0, 2024-03-28T18:50:11.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_notify": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_other": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_active": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_send": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_resend": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_ping": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_active": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_send": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_resend": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_active": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_send": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_resend": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_active": 0, 2024-03-28T18:50:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_send": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_resend": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "command_active": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "command_send": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "command_resend": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "map_epoch": 95, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "map_full": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "map_inc": 4, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_sessions": 9, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_open": 9, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_close": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_laggy": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_wr": 14, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_rd": 0, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_del": 1 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.050 INFO:teuthology.orchestra.run.smithi164.stdout: "oft": { 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_objs": 1, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_updates": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_removes": 0 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "osd-slow-ops": { 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_ops_count": 0 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "purge_queue": { 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed": 0, 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:11.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 637, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 829745, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 637, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 637, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 829745, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.052 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 37, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 11108, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 37, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 11108, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.053 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 1024, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 37, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 37, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 37, 2024-03-28T18:50:11.054 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 37, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:11.055 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle-0x55c1bfa00500": { 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 3, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 896, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 3, 2024-03-28T18:50:11.056 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 896, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:11.057 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-28T18:50:11.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:11 smithi192 ceph-mon[36589]: pgmap v723: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 60 MiB/s wr, 313 op/s 2024-03-28T18:50:11.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:11 smithi164 ceph-mon[36248]: pgmap v723: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 60 MiB/s wr, 313 op/s 2024-03-28T18:50:11.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:11 smithi145 ceph-mon[32151]: pgmap v723: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 60 MiB/s wr, 313 op/s 2024-03-28T18:50:12.830 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T18:50:12.830 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T18:50:12.830 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T18:50:12.831 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T18:50:13.088 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:50:13.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:13 smithi164 ceph-mon[36248]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 54 MiB/s wr, 159 op/s 2024-03-28T18:50:13.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:13 smithi192 ceph-mon[36589]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 54 MiB/s wr, 159 op/s 2024-03-28T18:50:13.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:13 smithi145 ceph-mon[32151]: pgmap v724: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 54 MiB/s wr, 159 op/s 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 1606, 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 1606, 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 473921, 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 509885, 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 7, 2024-03-28T18:50:15.096 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.155111331, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.043990853, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.041918309, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.058326414, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 1605, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.068413182, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000042625 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 262, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 252, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 807214, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 146846, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 4, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.038572512, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.013370785, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.021385448, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.006421967, 2024-03-28T18:50:15.098 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 251, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.014448386, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000057563 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 805824, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 145824 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:15.099 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 801, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 982, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 210222, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 641714, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 5, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 0.073385936, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 0.029025060, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.017390836, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.020676438, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 980, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.041131613, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000041971 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:15.100 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_encrypted_bytes": 387632 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "cct": { 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "total_workers": 1, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "unhealthy_workers": 0 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-MDSRank": { 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 4053, 2024-03-28T18:50:15.101 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.114808551, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000028326 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": { 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "request": 0, 2024-03-28T18:50:15.102 INFO:teuthology.orchestra.run.smithi164.stdout: "reply": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "reply_latency": { 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_reply": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "forward": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_commit": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_split": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_merge": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_top": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_bottom": 0, 2024-03-28T18:50:15.103 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pinned": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_expired": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_with_caps": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "subtrees": 1, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_hit": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_forward": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_discover": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_lock": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "load_cent": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "q": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "exported": 0, 2024-03-28T18:50:15.104 INFO:teuthology.orchestra.run.smithi164.stdout: "exported_inodes": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "imported": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "imported_inodes": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rfiles": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rbytes": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rsnaps": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:15.105 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "process_request_cap_release": 0 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_cache": { 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update": 0, 2024-03-28T18:50:15.106 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_try_discover": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_send_discover": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_created": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_enqueued": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_migrated": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_started": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_completed": 0, 2024-03-28T18:50:15.107 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_flush": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_inodestats": 0 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_log": { 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evlrg": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evadd": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evex": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evtrm": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "ev": 3, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evexg": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "evexd": 0, 2024-03-28T18:50:15.108 INFO:teuthology.orchestra.run.smithi164.stdout: "segadd": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "segex": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "segtrm": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "seg": 2, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "segmjr": 2, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "segexg": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "segexd": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "expos": 4194304, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "wrpos": 4195200, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "rdpos": 4194892, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "jlat": { 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.109 INFO:teuthology.orchestra.run.smithi164.stdout: "replayed": 3 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_mem": { 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "ino": 3, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "ino+": 3, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "ino-": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dir": 1, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dir+": 1, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dir-": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dn": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dn+": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "dn-": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "cap": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "cap+": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "cap-": 0, 2024-03-28T18:50:15.110 INFO:teuthology.orchestra.run.smithi164.stdout: "rss": 46400, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "heap": 215420 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_server": { 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_client_request": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_request": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_session": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_peer_request": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "req_create_latency": { 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getattr_latency": { 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.111 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "req_link_latency": { 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookup_latency": { 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.112 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupino_latency": { 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupname_latency": { 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lssnap_latency": { 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mkdir_latency": { 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mknod_latency": { 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mksnap_latency": { 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "req_open_latency": { 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "req_readdir_latency": { 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.116 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rename_latency": { 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.117 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmdir_latency": { 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.118 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setattr_latency": { 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.119 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setlayout_latency": { 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setxattr_latency": { 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "req_symlink_latency": { 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.120 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "req_unlink_latency": { 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.121 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_sessions": { 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "session_count": 1, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "session_add": 1, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "session_remove": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_open": 1, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_stale": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "total_load": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "average_load": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "avg_session_uptime": 2042, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "mempool": { 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:15.122 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:15.123 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_items": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:15.124 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_bytes": 205583, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_items": 59, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_items": 172, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:15.125 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_items": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_bytes": 8320, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_items": 13, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_items": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_items": 0 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "objecter": { 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "op_active": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "op_laggy": 0, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send": 2404, 2024-03-28T18:50:15.126 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send_bytes": 21, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_resend": 3, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_reply": 2404, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_latency": { 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2404, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 4.919919043, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.002046555 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_inflight": 0, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "oplen_avg": { 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 2404, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 2405 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op": 2404, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_r": 2404, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_w": 0, 2024-03-28T18:50:15.127 INFO:teuthology.orchestra.run.smithi164.stdout: "op_rmw": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "op_pg": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_stat": 1558, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_create": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_read": 844, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_write": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writefull": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writesame": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_append": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_zero": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_truncate": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_delete": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_mapext": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_getxattr": 1, 2024-03-28T18:50:15.128 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_setxattr": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_call": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_watch": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_notify": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_other": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_active": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_send": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_resend": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_ping": 0, 2024-03-28T18:50:15.129 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_active": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_send": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_resend": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_active": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_send": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_resend": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_active": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_send": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_resend": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "command_active": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "command_send": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "command_resend": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "map_epoch": 95, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "map_full": 0, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "map_inc": 4, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_sessions": 7, 2024-03-28T18:50:15.130 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_open": 7, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_close": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_laggy": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_wr": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_rd": 2, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_del": 0 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "oft": { 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_objs": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_updates": 0, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_total_removes": 0 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "osd-slow-ops": { 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "slow_ops_count": 0 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.131 INFO:teuthology.orchestra.run.smithi164.stdout: "purge_queue": { 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 2665, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 1235204, 2024-03-28T18:50:15.132 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 2665, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2665, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 1235204, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:15.133 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 2404, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 901, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2404, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 901, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:15.134 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 1024, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 2404, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 2404, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 2404, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 2404, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.135 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:15.136 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle-0x55e61e1fa6e0": { 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 0, 2024-03-28T18:50:15.137 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 0, 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-28T18:50:15.138 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-28T18:50:15.138 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:15 smithi164 ceph-mon[36248]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 58 MiB/s wr, 125 op/s 2024-03-28T18:50:15.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:15 smithi192 ceph-mon[36589]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 58 MiB/s wr, 125 op/s 2024-03-28T18:50:15.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:15 smithi145 ceph-mon[32151]: pgmap v725: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 58 MiB/s wr, 125 op/s 2024-03-28T18:50:15.915 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-28T18:50:15.915 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-28T18:50:15.915 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T18:50:15.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T18:50:16.558 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:50:17.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:17 smithi192 ceph-mon[36589]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 66 MiB/s wr, 122 op/s 2024-03-28T18:50:17.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:17 smithi164 ceph-mon[36248]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 66 MiB/s wr, 122 op/s 2024-03-28T18:50:17.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:17 smithi145 ceph-mon[32151]: pgmap v726: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 66 MiB/s wr, 122 op/s 2024-03-28T18:50:18.649 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-28T18:50:18.649 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 46, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 51, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 10679, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 60319, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 14, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 5, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.011361639, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.002433983, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.006955325, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000396378, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 49, 2024-03-28T18:50:18.650 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.012702922, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000259243 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 45, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000011832, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000262 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1936, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 1584 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 21, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 199, 2024-03-28T18:50:18.651 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 7558, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 404157, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.011495976, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.007229724, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.002561273, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000502118, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 198, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.023541323, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000118895 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.652 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 390928 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 1413, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 1052, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1093860, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 446785, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 5, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.121572054, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.048023177, 2024-03-28T18:50:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.092667949, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.010361341, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1051, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.122721600, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000116766 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1747, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000582362, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000333 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 818400, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 150000 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 51, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.001636368, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000032085 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000005101, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000002550 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 5, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 2, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.002703610, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001351805 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 3, 2024-03-28T18:50:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 11, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 14, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 14, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 14, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 1, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 1, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 2, 2024-03-28T18:50:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 8, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 3, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 2, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 22, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 10738150859, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 2, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 3, 2024-03-28T18:50:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 2, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 3 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 3, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-28T18:50:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 5, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 5, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 2, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 2, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-28T18:50:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 4195732, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 0, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.006919387, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001729846 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 0 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 17, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 17, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 0, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 15, 2024-03-28T18:50:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 16, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 1, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 14, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 14, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 0, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 1, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 2, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 1, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 50304, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 223612 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 7, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 5, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 14, 2024-03-28T18:50:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.002703610, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001351805 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 2, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 1, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 1, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 1, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 724, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-28T18:50:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 247511, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 87, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 172, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 66981, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 292, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 42, 2024-03-28T18:50:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 12135, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 33, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 42, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 42, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.212995874, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.005071330 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 42, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 66 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 42, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 0, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 42, 2024-03-28T18:50:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 10, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 4, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 7, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 6, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 0, 2024-03-28T18:50:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 20, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-28T18:50:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 95, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 4, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 8, 2024-03-28T18:50:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 8, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 18, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 0, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 1 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 1, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-28T18:50:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 1476, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 961173, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 1476, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 1476, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 961173, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 42, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 12135, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 42, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 12135, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 42, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 42, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 42, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 42, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x55ad31bfa500": { 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 5, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 1428, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 5, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 4, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 1428, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:18.691 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 MiB/s wr, 78 op/s 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:19 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 MiB/s wr, 78 op/s 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:19 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: pgmap v727: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 MiB/s wr, 78 op/s 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:19 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:19.557 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T18:50:19.557 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T18:50:19.557 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T18:50:19.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T18:50:19.821 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:50:20.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:20.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:20.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:20.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:20 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:20.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:20 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:20.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:20.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:20 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' 2024-03-28T18:50:21.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:21 smithi192 ceph-mon[36589]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 65 MiB/s wr, 93 op/s 2024-03-28T18:50:21.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:21 smithi164 ceph-mon[36248]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 65 MiB/s wr, 93 op/s 2024-03-28T18:50:21.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:21 smithi145 ceph-mon[32151]: pgmap v728: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 65 MiB/s wr, 93 op/s 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 3, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 4, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 2688, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 448, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 4, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 0, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.002183889, 2024-03-28T18:50:22.416 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.000048689, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.000791060, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.002123804, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000707934 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 2688, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.417 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 1, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 106, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 160, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 96176, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 3, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.005834716, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.004085975, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.000611159, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 104, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.009870766, 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000094911 2024-03-28T18:50:22.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 96176 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 141, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 140, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 179664, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 83392, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 2, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 1, 2024-03-28T18:50:22.419 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.024411211, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.009817256, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.009669222, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.004284420, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 139, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.020275664, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000145868 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 179664, 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 83392 2024-03-28T18:50:22.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:22.421 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:22.422 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 148945, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 29, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-28T18:50:22.423 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 928, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 2, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:22.424 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 142, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 159764, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 142, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 142, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 159764, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:22.425 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:22.426 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:22.426 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:22.426 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:22.426 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-28T18:50:23.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:23 smithi192 ceph-mon[36589]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 76 op/s 2024-03-28T18:50:23.234 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-03-28T18:50:23.234 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-03-28T18:50:23.234 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T18:50:23.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T18:50:23.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:23 smithi164 ceph-mon[36248]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 76 op/s 2024-03-28T18:50:23.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:23 smithi145 ceph-mon[32151]: pgmap v729: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 76 op/s 2024-03-28T18:50:23.556 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:50:25.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:25 smithi192 ceph-mon[36589]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 55 MiB/s wr, 80 op/s 2024-03-28T18:50:25.317 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-28T18:50:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 4366, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 3986, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1137634, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 280931309, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 21, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 2.449262395, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 1.216797537, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.131947718, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.976656666, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3988, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.998555861, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000501142 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1544, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000474253, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000307 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 3603, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 4128, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1006721, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 351412184, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 10, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 7, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 2.741738376, 2024-03-28T18:50:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 1.606448459, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.105310196, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 1.000790319, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4126, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2.678741924, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000649234 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1630, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000425046, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000260 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 399664 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 433664, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 435770, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 142125667, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 311550855, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 91, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 10, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 25.033396806, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 9.249329439, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 46.527233075, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 2.106549985, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 435766, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 234.347438209, 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000537782 2024-03-28T18:50:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 632377, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.272822382, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000431 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 925568, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 191648 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-28T18:50:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4912, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 289.301909918, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.058896968 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1018, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.039483664, 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000038785 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 168, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 168, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 168, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.424319659, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.002525712 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 148, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-28T18:50:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 29, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 5, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 24, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 21, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 7, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 7, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 2, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 309, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 210, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-28T18:50:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 4, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 234, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-28T18:50:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 1, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 33, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 3166, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 430165, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 430118, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 7, 2024-03-28T18:50:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 85 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 6, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 5, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 22, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 17, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-28T18:50:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 431436, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 425451, 2024-03-28T18:50:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 423535, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 7901, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 12377, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 534, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 527, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 525, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 9, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 1, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 14, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 832796576, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 849468575, 2024-03-28T18:50:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 0, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4938, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 717.482637118, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.145298225 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 0 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 28, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 45, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 17, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 18, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 22, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 4, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 29, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 69, 2024-03-28T18:50:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 40, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 7, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 44, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 37, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 86756, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 231804 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 249, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 168, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 24, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 22, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.066155499, 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.003007068 2024-03-28T18:50:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 28, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.007060885, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000252174 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-28T18:50:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 62, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.066197392, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001067699 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000121474, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000121474 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-28T18:50:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 10, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.020752842, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.002075284 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000901639, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000450819 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-28T18:50:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 6, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.003413683, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000568947 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.001964028, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000982014 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.018618228, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.004654557 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.006540728, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001635182 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 9, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.023399979, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.002599997 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 18, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.209193282, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.011621849 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 4, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 3, 2024-03-28T18:50:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 44, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 44, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 730, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 299662, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 128, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-28T18:50:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 172, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 110735, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 1275, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-28T18:50:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 7736, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 845636042, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 37, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 7736, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 7736, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 651.858582691, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.084263001 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-28T18:50:25.347 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 7736, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 8135 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 7736, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 5, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 7731, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 126, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 66, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 5, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 4943, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 156, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-28T18:50:25.348 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 1964, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 110, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-28T18:50:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 95, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 9, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 12, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 12, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 749, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 0, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 16 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-28T18:50:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 1, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 516, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 5, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 1054, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 17, 2024-03-28T18:50:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 441629, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 104494775, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 441629, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 441629, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 104494775, 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 7736, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 845637863, 2024-03-28T18:50:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 7736, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 845637863, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 7736, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 7736, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 7736, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 7736, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 17, 2024-03-28T18:50:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 1821, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 17, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 5, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 1821, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x5639c5acc500": { 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 431436, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 845274271, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 431436, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 4938, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 845274271, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:25.359 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-28T18:50:25.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:25 smithi164 ceph-mon[36248]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 55 MiB/s wr, 80 op/s 2024-03-28T18:50:25.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:25 smithi145 ceph-mon[32151]: pgmap v730: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 55 MiB/s wr, 80 op/s 2024-03-28T18:50:26.138 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T18:50:26.138 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T18:50:26.138 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T18:50:26.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid b991dbde-ed30-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T18:50:26.397 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:50:27.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:27 smithi192 ceph-mon[36589]: pgmap v731: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 61 MiB/s wr, 86 op/s 2024-03-28T18:50:27.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:27 smithi164 ceph-mon[36248]: pgmap v731: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 61 MiB/s wr, 86 op/s 2024-03-28T18:50:27.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:27 smithi145 ceph-mon[32151]: pgmap v731: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 61 MiB/s wr, 86 op/s 2024-03-28T18:50:28.588 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-28T18:50:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:50:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 814, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 816, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 215287, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 258916, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.065320929, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.017734705, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.017303500, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.021894373, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 814, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.051923751, 2024-03-28T18:50:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000063788 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 744, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 928, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 184914, 2024-03-28T18:50:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 717025, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.079727263, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.021086932, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.018520010, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.032663523, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 927, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.048565350, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000052389 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 482336 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 1072, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 1062, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1079296, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 405600, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 3, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.119918093, 2024-03-28T18:50:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.034079631, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.043895880, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.038472238, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1061, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.067732580, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000063838 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 807808, 2024-03-28T18:50:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 148656 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3956, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.124629279, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000031503 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 0, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 0, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 0, 2024-03-28T18:50:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 1, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-28T18:50:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-28T18:50:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 0 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-28T18:50:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:50:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 5, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 0, 2024-03-28T18:50:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 2, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 4195732, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 4195424, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 5 2024-03-28T18:50:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 3, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 3, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 1, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 1, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 0, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 50564, 2024-03-28T18:50:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 215420 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-28T18:50:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-28T18:50:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 2, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 1, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 0, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 0, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 2066, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:50:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:50:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-28T18:50:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 190208, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 53, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 8984, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 172, 2024-03-28T18:50:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 8320, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 13, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 2360, 2024-03-28T18:50:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 21, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 3, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 2360, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2360, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 49.228063670, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.020859349 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2360, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2361 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 2360, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2360, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 0, 2024-03-28T18:50:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 1544, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 814, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-28T18:50:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 1, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-28T18:50:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 95, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 4, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 5, 2024-03-28T18:50:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 5, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 2, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 0 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-28T18:50:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 2626, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 1226564, 2024-03-28T18:50:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 2626, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2626, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 1226564, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2360, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 1433, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2360, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 1433, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-28T18:50:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2360, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 2360, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2360, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 2360, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x5601c85506e0": { 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-28T18:50:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-28T18:50:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-28T18:50:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-28T18:50:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-28T18:50:28.631 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-28T18:50:29.362 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:29 smithi192 ceph-mon[36589]: pgmap v732: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 51 MiB/s wr, 71 op/s 2024-03-28T18:50:29.364 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-28T18:50:29.364 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-28T18:50:29.365 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T18:50:29.375 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T18:50:29.376 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:50:29.376 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:50:29.398 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-28T18:50:29.398 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:50:29.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:50:29.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:29 smithi164 ceph-mon[36248]: pgmap v732: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 51 MiB/s wr, 71 op/s 2024-03-28T18:50:29.519 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:29 smithi145 ceph-mon[32151]: pgmap v732: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 51 MiB/s wr, 71 op/s 2024-03-28T18:50:29.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 -- 172.21.15.145:0/997141181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef881017b0 msgr2=0x7fef88101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:29.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/997141181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef881017b0 0x7fef88101c30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef78009900 tx=0x7fef7802bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:29.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 -- 172.21.15.145:0/997141181 shutdown_connections 2024-03-28T18:50:29.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/997141181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef88102170 0x7fef88108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.603 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/997141181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef881017b0 0x7fef88101c30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/997141181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881005c0 0x7fef881009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.600+0000 7fef8fa0b640 1 -- 172.21.15.145:0/997141181 >> 172.21.15.145:0/997141181 conn(0x7fef880fbb30 msgr2=0x7fef880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- 172.21.15.145:0/997141181 shutdown_connections 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- 172.21.15.145:0/997141181 wait complete. 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 Processor -- start 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- start start 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef881005c0 0x7fef88197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881017b0 0x7fef88197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8810c240 con 0x7fef88102170 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8810c0c0 con 0x7fef881005c0 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8fa0b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8810c3c0 con 0x7fef881017b0 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8dfa8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:29.604 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8dfa8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40048/0 (socket says 172.21.15.145:40048) 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8dfa8640 1 -- 172.21.15.145:0/2167586487 learned_addr learned my addr 172.21.15.145:0/2167586487 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8d7a7640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef881005c0 0x7fef88197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.601+0000 7fef8cfa6640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881017b0 0x7fef88197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 -- 172.21.15.145:0/2167586487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881017b0 msgr2=0x7fef88197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881017b0 0x7fef88197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 -- 172.21.15.145:0/2167586487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef881005c0 msgr2=0x7fef88197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef881005c0 0x7fef88197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 -- 172.21.15.145:0/2167586487 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8819f610 con 0x7fef88102170 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8dfa8640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef8000eb10 tx=0x7fef8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80004b20 con 0x7fef88102170 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef80004cc0 con 0x7fef88102170 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80010760 con 0x7fef88102170 2024-03-28T18:50:29.605 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8819f900 con 0x7fef88102170 2024-03-28T18:50:29.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.602+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8819fdd0 con 0x7fef88102170 2024-03-28T18:50:29.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.603+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef881a0090 con 0x7fef88102170 2024-03-28T18:50:29.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.604+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fef80005020 con 0x7fef88102170 2024-03-28T18:50:29.606 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.605+0000 7fef7e7fc640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:29.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.605+0000 7fef8d7a7640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:29.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.605+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fef800cf380 con 0x7fef88102170 2024-03-28T18:50:29.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.605+0000 7fef8d7a7640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 0x7fef740a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef88101610 tx=0x7fef84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:29.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.610+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef8009c070 con 0x7fef88102170 2024-03-28T18:50:29.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.867+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef881a0260 con 0x7fef88102170 2024-03-28T18:50:29.869 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.868+0000 7fef7e7fc640 1 -- 172.21.15.145:0/2167586487 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7fef8009b970 con 0x7fef88102170 2024-03-28T18:50:29.869 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:50:29.869 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 msgr2=0x7fef740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 0x7fef740a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef88101610 tx=0x7fef84005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 msgr2=0x7fef8819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef8000eb10 tx=0x7fef8000c2d0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 shutdown_connections 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef88102170 0x7fef8819efa0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.870+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.871+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef881017b0 0x7fef88197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.871+0000 7fef8fa0b640 1 --2- 172.21.15.145:0/2167586487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef881005c0 0x7fef88197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:29.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.871+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 >> 172.21.15.145:0/2167586487 conn(0x7fef880fbb30 msgr2=0x7fef880fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:29.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.871+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 shutdown_connections 2024-03-28T18:50:29.872 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:29.871+0000 7fef8fa0b640 1 -- 172.21.15.145:0/2167586487 wait complete. 2024-03-28T18:50:29.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:31:13.733117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:35:48.277653+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:35:49.458851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T18:35:53.237697+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T18:50:29.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- 172.21.15.145:0/1273087091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0101790 msgr2=0x7f52d0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 --2- 172.21.15.145:0/1273087091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0101790 0x7f52d0101c10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009900 tx=0x7f52c402bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- 172.21.15.145:0/1273087091 shutdown_connections 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 --2- 172.21.15.145:0/1273087091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d0102150 0x7f52d01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 --2- 172.21.15.145:0/1273087091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0101790 0x7f52d0101c10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 --2- 172.21.15.145:0/1273087091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d01005a0 0x7f52d01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- 172.21.15.145:0/1273087091 >> 172.21.15.145:0/1273087091 conn(0x7f52d00fbb30 msgr2=0x7f52d00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- 172.21.15.145:0/1273087091 shutdown_connections 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- 172.21.15.145:0/1273087091 wait complete. 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 Processor -- start 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.031+0000 7f52d64cc640 1 -- start start 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 0x7f52d01968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d0101790 0x7f52d0196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d010c230 con 0x7f52d0102150 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d010c0b0 con 0x7f52d0101790 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d64cc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d010c3b0 con 0x7f52d01005a0 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 0x7f52d01968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52cf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d0101790 0x7f52d0196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.034 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40056/0 (socket says 172.21.15.145:40056) 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52cffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 0x7f52d01968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:36218/0 (socket says 172.21.15.145:36218) 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52cffff640 1 -- 172.21.15.145:0/2988503550 learned_addr learned my addr 172.21.15.145:0/2988503550 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 -- 172.21.15.145:0/2988503550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 msgr2=0x7f52d01968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 0x7f52d01968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 -- 172.21.15.145:0/2988503550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d0101790 msgr2=0x7f52d0196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d0101790 0x7f52d0196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 -- 172.21.15.145:0/2988503550 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d019e4e0 con 0x7f52d0102150 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52d4a69640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009fd0 tx=0x7f52c800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.032+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c800ce20 con 0x7f52d0102150 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.033+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d019e7d0 con 0x7f52d0102150 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.033+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52c8005020 con 0x7f52d0102150 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.033+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8010720 con 0x7f52d0102150 2024-03-28T18:50:30.035 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.033+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d019ed40 con 0x7f52d0102150 2024-03-28T18:50:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.034+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0107940 con 0x7f52d0102150 2024-03-28T18:50:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.035+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f52c8005540 con 0x7f52d0102150 2024-03-28T18:50:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.035+0000 7f52cd7fa640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 0x7f52b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.035+0000 7f52cffff640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 0x7f52b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.036 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.035+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f52c80cf380 con 0x7f52d0102150 2024-03-28T18:50:30.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.036+0000 7f52cffff640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 0x7f52b80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f52c00095f0 tx=0x7f52c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.040+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f52c809c110 con 0x7f52d0102150 2024-03-28T18:50:30.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.228+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52d002d070 con 0x7f52d0102150 2024-03-28T18:50:30.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.228+0000 7f52cd7fa640 1 -- 172.21.15.145:0/2988503550 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52c809ba10 con 0x7f52d0102150 2024-03-28T18:50:30.228 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 1 2024-03-28T18:50:30.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 msgr2=0x7f52b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 0x7f52b80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f52c00095f0 tx=0x7f52c0005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 msgr2=0x7f52d019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009fd0 tx=0x7f52c800ef10 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 shutdown_connections 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52d0102150 0x7f52d019de70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d0101790 0x7f52d0196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f52b80a2800 0x7f52b80a4cc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 --2- 172.21.15.145:0/2988503550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52d01005a0 0x7f52d01968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 >> 172.21.15.145:0/2988503550 conn(0x7f52d00fbb30 msgr2=0x7f52d0104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 shutdown_connections 2024-03-28T18:50:30.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.231+0000 7f52d64cc640 1 -- 172.21.15.145:0/2988503550 wait complete. 2024-03-28T18:50:30.242 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:30 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2167586487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:30.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- 172.21.15.145:0/2304923714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1017a0 msgr2=0x7fbf6c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- 172.21.15.145:0/2304923714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c101c20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009900 tx=0x7fbf6002bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- 172.21.15.145:0/2304923714 shutdown_connections 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- 172.21.15.145:0/2304923714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 0x7fbf6c1089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- 172.21.15.145:0/2304923714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c101c20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- 172.21.15.145:0/2304923714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- 172.21.15.145:0/2304923714 >> 172.21.15.145:0/2304923714 conn(0x7fbf6c0fbb40 msgr2=0x7fbf6c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- 172.21.15.145:0/2304923714 shutdown_connections 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- 172.21.15.145:0/2304923714 wait complete. 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 Processor -- start 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 -- start start 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.350+0000 7fbf73590640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 0x7fbf6c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10c230 con 0x7fbf6c1005b0 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10c0b0 con 0x7fbf6c102160 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10c3b0 con 0x7fbf6c1017a0 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40076/0 (socket says 172.21.15.145:40076) 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 -- 172.21.15.145:0/39321899 learned_addr learned my addr 172.21.15.145:0/39321899 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf70b2b640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf71b2d640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 0x7fbf6c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 -- 172.21.15.145:0/39321899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 msgr2=0x7fbf6c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 -- 172.21.15.145:0/39321899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 msgr2=0x7fbf6c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 0x7fbf6c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 -- 172.21.15.145:0/39321899 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c19f650 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf70b2b640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf7132c640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00eb70 tx=0x7fbf5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c003b70 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf5c00ee70 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c19f940 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c010830 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.351+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c19fe10 con 0x7fbf6c1005b0 2024-03-28T18:50:30.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.352+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c0688e0 con 0x7fbf6c1005b0 2024-03-28T18:50:30.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.354+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbf5c004a40 con 0x7fbf6c1005b0 2024-03-28T18:50:30.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.355+0000 7fbf6a7fc640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 0x7fbf540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.355+0000 7fbf70b2b640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 0x7fbf540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.355+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbf5c0cf5b0 con 0x7fbf6c1005b0 2024-03-28T18:50:30.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.355+0000 7fbf70b2b640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 0x7fbf540a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf6002c7b0 tx=0x7fbf60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.358 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.357+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf5c09c2a0 con 0x7fbf6c1005b0 2024-03-28T18:50:30.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:30 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2167586487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:30.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:30 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2167586487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:30.540 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.539+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf6c02d200 con 0x7fbf6c1005b0 2024-03-28T18:50:30.540 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.540+0000 7fbf6a7fc640 1 -- 172.21.15.145:0/39321899 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf5c09bba0 con 0x7fbf6c1005b0 2024-03-28T18:50:30.540 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 msgr2=0x7fbf540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 0x7fbf540a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf6002c7b0 tx=0x7fbf60005fd0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 msgr2=0x7fbf6c06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00eb70 tx=0x7fbf5c00c6a0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 shutdown_connections 2024-03-28T18:50:30.543 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf6c102160 0x7fbf6c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf540a2800 0x7fbf540a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf6c1017a0 0x7fbf6c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 --2- 172.21.15.145:0/39321899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf6c1005b0 0x7fbf6c06a7e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 >> 172.21.15.145:0/39321899 conn(0x7fbf6c0fbb40 msgr2=0x7fbf6c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 shutdown_connections 2024-03-28T18:50:30.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.543+0000 7fbf73590640 1 -- 172.21.15.145:0/39321899 wait complete. 2024-03-28T18:50:30.554 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 -- 172.21.15.145:0/404575682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff901028a0 msgr2=0x7eff90102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 --2- 172.21.15.145:0/404575682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff901028a0 0x7eff90102d20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7eff80009900 tx=0x7eff8002bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 -- 172.21.15.145:0/404575682 shutdown_connections 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 --2- 172.21.15.145:0/404575682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff90103260 0x7eff90109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 --2- 172.21.15.145:0/404575682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff901028a0 0x7eff90102d20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 --2- 172.21.15.145:0/404575682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff90101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 -- 172.21.15.145:0/404575682 >> 172.21.15.145:0/404575682 conn(0x7eff900fcc40 msgr2=0x7eff900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 -- 172.21.15.145:0/404575682 shutdown_connections 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.663+0000 7eff972b2640 1 -- 172.21.15.145:0/404575682 wait complete. 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 Processor -- start 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 -- start start 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9010d330 con 0x7eff90103260 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9010d1b0 con 0x7eff901016b0 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff972b2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9010d4b0 con 0x7eff901028a0 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9584f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9504e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.145:36266/0 (socket says 172.21.15.145:36266) 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9584f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40102/0 (socket says 172.21.15.145:40102) 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 -- 172.21.15.145:0/1609404574 learned_addr learned my addr 172.21.15.145:0/1609404574 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9504e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:52048/0 (socket says 172.21.15.145:52048) 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 -- 172.21.15.145:0/1609404574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 msgr2=0x7eff901979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 -- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 msgr2=0x7eff9019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.664+0000 7eff9484d640 1 -- 172.21.15.145:0/1609404574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9019f600 con 0x7eff901028a0 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff9584f640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff9504e640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff9484d640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff80009230 tx=0x7eff8002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80041070 con 0x7eff901028a0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9019f890 con 0x7eff901028a0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff80004450 con 0x7eff901028a0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80003e20 con 0x7eff901028a0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.665+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9019fdd0 con 0x7eff901028a0 2024-03-28T18:50:30.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.666+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90108a50 con 0x7eff901028a0 2024-03-28T18:50:30.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.667+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7eff80044070 con 0x7eff901028a0 2024-03-28T18:50:30.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.668+0000 7eff867fc640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 0x7eff7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.668+0000 7eff9504e640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 0x7eff7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.668+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7eff800f0d10 con 0x7eff901028a0 2024-03-28T18:50:30.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.668+0000 7eff9504e640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 0x7eff7c0a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7eff90102700 tx=0x7eff88009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.674+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff800bd3b0 con 0x7eff901028a0 2024-03-28T18:50:30.854 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.854+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff9002d070 con 0x7eff901028a0 2024-03-28T18:50:30.855 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.855+0000 7eff867fc640 1 -- 172.21.15.145:0/1609404574 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff800c1450 con 0x7eff901028a0 2024-03-28T18:50:30.855 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 64 2024-03-28T18:50:30.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 msgr2=0x7eff7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 0x7eff7c0a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7eff90102700 tx=0x7eff88009500 comp rx=0 tx=0).stop 2024-03-28T18:50:30.857 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 msgr2=0x7eff90197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff80009230 tx=0x7eff8002c650 comp rx=0 tx=0).stop 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 shutdown_connections 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff90103260 0x7eff9019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7eff7c0a2800 0x7eff7c0a4cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff901028a0 0x7eff90197f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 --2- 172.21.15.145:0/1609404574 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff901016b0 0x7eff901979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 >> 172.21.15.145:0/1609404574 conn(0x7eff900fcc40 msgr2=0x7eff90105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 shutdown_connections 2024-03-28T18:50:30.858 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.857+0000 7eff972b2640 1 -- 172.21.15.145:0/1609404574 wait complete. 2024-03-28T18:50:30.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T18:50:30.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.990+0000 7f147ed37640 1 -- 172.21.15.145:0/3277103196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 msgr2=0x7f1478108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.990+0000 7f147ed37640 1 --2- 172.21.15.145:0/3277103196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478108a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1468009860 tx=0x7f146802bb70 comp rx=0 tx=0).stop 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- 172.21.15.145:0/3277103196 shutdown_connections 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- 172.21.15.145:0/3277103196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478108a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- 172.21.15.145:0/3277103196 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781017b0 0x7f1478101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- 172.21.15.145:0/3277103196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781005c0 0x7f14781009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- 172.21.15.145:0/3277103196 >> 172.21.15.145:0/3277103196 conn(0x7f14780fbb50 msgr2=0x7f14780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- 172.21.15.145:0/3277103196 shutdown_connections 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- 172.21.15.145:0/3277103196 wait complete. 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 Processor -- start 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- start start 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 0x7f14781046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.994 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478103220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c240 con 0x7f14781017b0 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c0c0 con 0x7f14781005c0 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147ed37640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810c3c0 con 0x7f1478102170 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f1477fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f1477fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40128/0 (socket says 172.21.15.145:40128) 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f1477fff640 1 -- 172.21.15.145:0/3798345566 learned_addr learned my addr 172.21.15.145:0/3798345566 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147cad3640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 0x7f14781046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.991+0000 7f147d2d4640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478103220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 -- 172.21.15.145:0/3798345566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 msgr2=0x7f1478103220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478103220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 -- 172.21.15.145:0/3798345566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 msgr2=0x7f14781046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:30.995 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 0x7f14781046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 -- 172.21.15.145:0/3798345566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478103ab0 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f147d2d4640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f147cad3640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 0x7f14781046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1477fff640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f146000e9f0 tx=0x7f146000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146000cd80 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14781a1a80 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1460005500 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460010690 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.992+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781a1fd0 con 0x7f14781017b0 2024-03-28T18:50:30.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.993+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14781a2260 con 0x7f14781017b0 2024-03-28T18:50:30.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.994+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f14600046c0 con 0x7f14781017b0 2024-03-28T18:50:30.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.995+0000 7f1475ffb640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 0x7f14640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:30.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.995+0000 7f147cad3640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 0x7f14640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:30.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.995+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f14600cf470 con 0x7f14781017b0 2024-03-28T18:50:30.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.996+0000 7f147cad3640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 0x7f14640a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1478101610 tx=0x7f1470010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:30.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:30.999+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1460053740 con 0x7f14781017b0 2024-03-28T18:50:31.181 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.181+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f14781a2430 con 0x7f14781017b0 2024-03-28T18:50:31.182 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.181+0000 7f1475ffb640 1 -- 172.21.15.145:0/3798345566 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7f146009c160 con 0x7f14781017b0 2024-03-28T18:50:31.182 INFO:teuthology.orchestra.run.smithi145.stdout:pg_num: 32 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 msgr2=0x7f14640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 0x7f14640a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1478101610 tx=0x7f1470010040 comp rx=0 tx=0).stop 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 msgr2=0x7f1478102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f146000e9f0 tx=0x7f146000edf0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 shutdown_connections 2024-03-28T18:50:31.184 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f14640a2800 0x7f14640a4cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1478102170 0x7f1478103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14781017b0 0x7f1478102ce0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 --2- 172.21.15.145:0/3798345566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14781005c0 0x7f14781046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 >> 172.21.15.145:0/3798345566 conn(0x7f14780fbb50 msgr2=0x7f1478069cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 shutdown_connections 2024-03-28T18:50:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.184+0000 7f147ed37640 1 -- 172.21.15.145:0/3798345566 wait complete. 2024-03-28T18:50:31.195 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:31 smithi145 ceph-mon[32151]: pgmap v733: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 89 op/s 2024-03-28T18:50:31.195 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:31 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2988503550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.195 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:31 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/39321899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.195 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:31 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1609404574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T18:50:31.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 -- 172.21.15.145:0/437484853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c641005a0 msgr2=0x7f6c641009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 --2- 172.21.15.145:0/437484853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c641005a0 0x7f6c641009c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009900 tx=0x7f6c5402bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:31.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 -- 172.21.15.145:0/437484853 shutdown_connections 2024-03-28T18:50:31.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 --2- 172.21.15.145:0/437484853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c64102150 0x7f6c641089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.304 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 --2- 172.21.15.145:0/437484853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64101790 0x7f6c64101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 --2- 172.21.15.145:0/437484853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c641005a0 0x7f6c641009c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 -- 172.21.15.145:0/437484853 >> 172.21.15.145:0/437484853 conn(0x7f6c640fbb30 msgr2=0x7f6c640fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 -- 172.21.15.145:0/437484853 shutdown_connections 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.301+0000 7f6c68863640 1 -- 172.21.15.145:0/437484853 wait complete. 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 Processor -- start 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 -- start start 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 0x7f6c64196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64102150 0x7f6c6419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6410c240 con 0x7f6c64101790 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6410c0c0 con 0x7f6c64102150 2024-03-28T18:50:31.305 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c68863640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6410c3c0 con 0x7f6c641005a0 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:40148/0 (socket says 172.21.15.145:40148) 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 -- 172.21.15.145:0/3186098144 learned_addr learned my addr 172.21.15.145:0/3186098144 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c6259c640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 0x7f6c64196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c62d9d640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64102150 0x7f6c6419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 -- 172.21.15.145:0/3186098144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 msgr2=0x7f6c64196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 0x7f6c64196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 -- 172.21.15.145:0/3186098144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64102150 msgr2=0x7f6c6419df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64102150 0x7f6c6419df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 -- 172.21.15.145:0/3186098144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6419e580 con 0x7f6c64101790 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c6259c640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 0x7f6c64196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.302+0000 7f6c61d9b640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800e9f0 tx=0x7f6c5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.303+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800ccc0 con 0x7f6c64101790 2024-03-28T18:50:31.306 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.303+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c58010910 con 0x7f6c64101790 2024-03-28T18:50:31.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.303+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6419e870 con 0x7f6c64101790 2024-03-28T18:50:31.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.303+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c580188a0 con 0x7f6c64101790 2024-03-28T18:50:31.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.303+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6419ee10 con 0x7f6c64101790 2024-03-28T18:50:31.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.304+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c64107990 con 0x7f6c64101790 2024-03-28T18:50:31.307 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.305+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6c58010ac0 con 0x7f6c64101790 2024-03-28T18:50:31.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.305+0000 7f6c537fe640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 0x7f6c4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:31.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.305+0000 7f6c6259c640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 0x7f6c4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:31.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.305+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6c580cf650 con 0x7f6c64101790 2024-03-28T18:50:31.308 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.306+0000 7f6c6259c640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 0x7f6c4c0a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6c5402c170 tx=0x7f6c5402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:31.309 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.309+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c5809c340 con 0x7f6c64101790 2024-03-28T18:50:31.398 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:31 smithi192 ceph-mon[36589]: pgmap v733: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 89 op/s 2024-03-28T18:50:31.398 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:31 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2988503550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.399 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:31 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/39321899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.399 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:31 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1609404574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:31 smithi164 ceph-mon[36248]: pgmap v733: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 89 op/s 2024-03-28T18:50:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:31 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2988503550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:31 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/39321899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:31 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1609404574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:50:31.511 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.511+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c6402d2c0 con 0x7f6c64101790 2024-03-28T18:50:31.511 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.511+0000 7f6c537fe640 1 -- 172.21.15.145:0/3186098144 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v95) v1 ==== 81+0+600 (secure 0 0 0) 0x7f6c5809bc40 con 0x7f6c64101790 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/606368710 2024-03-29T18:28:53.394017+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/4006486926 2024-03-29T18:28:53.394017+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/2249018729 2024-03-29T18:28:28.346767+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3480949122 2024-03-29T18:28:28.346767+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/832150839 2024-03-29T18:28:28.346767+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6800/2044834522 2024-03-29T18:28:28.346767+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:6801/2089479814 2024-03-29T18:28:53.394017+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145:0/3638297666 2024-03-29T18:28:53.394017+0000 2024-03-28T18:50:31.512 INFO:teuthology.orchestra.run.smithi145.stderr:listed 10 entries 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 msgr2=0x7f6c4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 0x7f6c4c0a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6c5402c170 tx=0x7f6c5402df90 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 msgr2=0x7f6c64196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800e9f0 tx=0x7f6c5800edf0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 shutdown_connections 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c64102150 0x7f6c6419df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c64101790 0x7f6c64196ea0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6c4c0a2880 0x7f6c4c0a4d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 --2- 172.21.15.145:0/3186098144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641005a0 0x7f6c64196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 >> 172.21.15.145:0/3186098144 conn(0x7f6c640fbb30 msgr2=0x7f6c64104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:31.514 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 shutdown_connections 2024-03-28T18:50:31.515 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:31.513+0000 7f6c68863640 1 -- 172.21.15.145:0/3186098144 wait complete. 2024-03-28T18:50:31.525 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T18:50:31.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:31.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T18:50:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:32 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3798345566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:50:32.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:32 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3186098144' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:50:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:32 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3798345566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:50:32.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:32 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3186098144' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:50:32.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:32 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3798345566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T18:50:32.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:32 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3186098144' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:50:33.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:33 smithi192 ceph-mon[36589]: pgmap v734: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 74 op/s 2024-03-28T18:50:33.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:33 smithi164 ceph-mon[36248]: pgmap v734: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 74 op/s 2024-03-28T18:50:33.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:33 smithi145 ceph-mon[32151]: pgmap v734: 129 pgs: 129 active+clean; 6.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 53 MiB/s wr, 74 op/s 2024-03-28T18:50:34.761 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T18:50:34.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:34.761 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T18:50:34.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:34.801 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-28T18:50:34.894 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T18:50:34.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:34.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:50:34.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:34.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-03-28T18:50:35.040 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T18:50:35.056 INFO:tasks.cephadm:Teardown begin 2024-03-28T18:50:35.056 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T18:50:35.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:50:35.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.226+0000 7f79091df640 1 -- 172.21.15.145:0/845998901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 msgr2=0x7f7904100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.226+0000 7f79091df640 1 --2- 172.21.15.145:0/845998901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904100ef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009900 tx=0x7f78f402bc60 comp rx=0 tx=0).stop 2024-03-28T18:50:35.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 -- 172.21.15.145:0/845998901 shutdown_connections 2024-03-28T18:50:35.228 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 --2- 172.21.15.145:0/845998901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 --2- 172.21.15.145:0/845998901 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f7904101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 --2- 172.21.15.145:0/845998901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904100ef0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 -- 172.21.15.145:0/845998901 >> 172.21.15.145:0/845998901 conn(0x7f79040fb2a0 msgr2=0x7f79040fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 -- 172.21.15.145:0/845998901 shutdown_connections 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.227+0000 7f79091df640 1 -- 172.21.15.145:0/845998901 wait complete. 2024-03-28T18:50:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.228+0000 7f79091df640 1 Processor -- start 2024-03-28T18:50:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 -- start start 2024-03-28T18:50:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790406b800 con 0x7f79041017e0 2024-03-28T18:50:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790406b680 con 0x7f79041021a0 2024-03-28T18:50:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.231+0000 7f79091df640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790406b980 con 0x7f7904100ad0 2024-03-28T18:50:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41738/0 (socket says 172.21.15.145:41738) 2024-03-28T18:50:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 -- 172.21.15.145:0/1263309894 learned_addr learned my addr 172.21.15.145:0/1263309894 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f7902d9d640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 -- 172.21.15.145:0/1263309894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 msgr2=0x7f7904070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 -- 172.21.15.145:0/1263309894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 msgr2=0x7f7904072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790259c640 1 -- 172.21.15.145:0/1263309894 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904072b70 con 0x7f79041017e0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.232+0000 7f790359e640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f790259c640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f78f800b4a0 tx=0x7f78f800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8014070 con 0x7f79041017e0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78f8010af0 con 0x7f79041017e0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f79091df640 1 -- 172.21.15.145:0/1263309894 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7904072e60 con 0x7f79041017e0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f79091df640 1 -- 172.21.15.145:0/1263309894 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7904073400 con 0x7f79041017e0 2024-03-28T18:50:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.233+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8017d10 con 0x7f79041017e0 2024-03-28T18:50:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.238+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f78f8017070 con 0x7f79041017e0 2024-03-28T18:50:35.240 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.239+0000 7f78ebfff640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 0x7f78ec0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.241 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.240+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f78f80ce030 con 0x7f79041017e0 2024-03-28T18:50:35.241 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.240+0000 7f7902d9d640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 0x7f78ec0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.242 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.241+0000 7f7902d9d640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 0x7f78ec0a4dc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f78f402c1a0 tx=0x7f78f4005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:35.246 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.245+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f00052c0 con 0x7f79041017e0 2024-03-28T18:50:35.256 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.254+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f78f809acc0 con 0x7f79041017e0 2024-03-28T18:50:35.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:35 smithi192 ceph-mon[36589]: pgmap v735: 129 pgs: 129 active+clean; 6.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 69 MiB/s wr, 257 op/s 2024-03-28T18:50:35.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:35 smithi164 ceph-mon[36248]: pgmap v735: 129 pgs: 129 active+clean; 6.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 69 MiB/s wr, 257 op/s 2024-03-28T18:50:35.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:35 smithi145 ceph-mon[32151]: pgmap v735: 129 pgs: 129 active+clean; 6.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 69 MiB/s wr, 257 op/s 2024-03-28T18:50:35.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.677+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f00055b0 con 0x7f79041017e0 2024-03-28T18:50:35.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.679+0000 7f78ebfff640 1 -- 172.21.15.145:0/1263309894 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11482 (secure 0 0 0) 0x7f78f809a5c0 con 0x7f79041017e0 2024-03-28T18:50:35.680 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:50:35.680 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":25603,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2973467272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2973467272},{"type":"v1","addr":"172.21.15.192:6835","nonce":2973467272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:41:47.139048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14556":{"gid":14556,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24461,24496,14631,24491,24469]},"id":1}]} 2024-03-28T18:50:35.681 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 31 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 msgr2=0x7f78ec0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 0x7f78ec0a4dc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f78f402c1a0 tx=0x7f78f4005b50 comp rx=0 tx=0).stop 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 msgr2=0x7f79040714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f78f800b4a0 tx=0x7f78f800b960 comp rx=0 tx=0).stop 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 shutdown_connections 2024-03-28T18:50:35.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79041021a0 0x7f7904072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79041017e0 0x7f79040714c0 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f78f800b4a0 tx=0x7f78f800b960 comp rx=0 tx=0).stop 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f78ec0a2900 0x7f78ec0a4dc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 --2- 172.21.15.145:0/1263309894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7904100ad0 0x7f7904070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.682+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 >> 172.21.15.145:0/1263309894 conn(0x7f79040fb2a0 msgr2=0x7f7904104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.683+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 shutdown_connections 2024-03-28T18:50:35.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.683+0000 7f78e9ffb640 1 -- 172.21.15.145:0/1263309894 wait complete. 2024-03-28T18:50:35.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:50:35.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.877+0000 7f6b05a1f640 1 -- 172.21.15.145:0/693141037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af80945f0 msgr2=0x7f6af8096a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.879 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.877+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/693141037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af80945f0 0x7f6af8096a20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6b00064b90 tx=0x7f6b000973c0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.881 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.880+0000 7f6b05a1f640 1 -- 172.21.15.145:0/693141037 shutdown_connections 2024-03-28T18:50:35.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.880+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/693141037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af8099910 0x7f6af8099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.880+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/693141037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6af8096f60 0x7f6af80993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.880+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/693141037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af80945f0 0x7f6af8096a20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.880+0000 7f6b05a1f640 1 -- 172.21.15.145:0/693141037 >> 172.21.15.145:0/693141037 conn(0x7f6af80901d0 msgr2=0x7f6af8092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:35.882 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.881+0000 7f6b05a1f640 1 -- 172.21.15.145:0/693141037 shutdown_connections 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.884+0000 7f6b05a1f640 1 -- 172.21.15.145:0/693141037 wait complete. 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 Processor -- start 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 -- start start 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af80945f0 0x7f6af81435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6af8096f60 0x7f6af8143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b9040 con 0x7f6af8099910 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b8ec0 con 0x7f6af8096f60 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.885+0000 7f6b05a1f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b91c0 con 0x7f6af80945f0 2024-03-28T18:50:35.887 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.886+0000 7f6b0521e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.890 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.886+0000 7f6b0521e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41742/0 (socket says 172.21.15.145:41742) 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.886+0000 7f6b0521e640 1 -- 172.21.15.145:0/2314501623 learned_addr learned my addr 172.21.15.145:0/2314501623 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.886+0000 7f6b04a1d640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af80945f0 0x7f6af81435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 -- 172.21.15.145:0/2314501623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af80945f0 msgr2=0x7f6af81435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af80945f0 0x7f6af81435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 -- 172.21.15.145:0/2314501623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6af8096f60 msgr2=0x7f6af8143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6af8096f60 0x7f6af8143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 -- 172.21.15.145:0/2314501623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af80ae0f0 con 0x7f6af8099910 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.887+0000 7f6b0521e640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6af000b3f0 tx=0x7f6af000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.888+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0015020 con 0x7f6af8099910 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.888+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af80ae3e0 con 0x7f6af8099910 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.888+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af80ae9b0 con 0x7f6af8099910 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.889+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6af0011d10 con 0x7f6af8099910 2024-03-28T18:50:35.891 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.889+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000f2e0 con 0x7f6af8099910 2024-03-28T18:50:35.892 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.890+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af8091cc0 con 0x7f6af8099910 2024-03-28T18:50:35.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.894+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6af0018020 con 0x7f6af8099910 2024-03-28T18:50:35.895 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.895+0000 7f6afdffb640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 0x7f6ae80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:35.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.895+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6af00cebf0 con 0x7f6af8099910 2024-03-28T18:50:35.896 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.895+0000 7f6b04a1d640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 0x7f6ae80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:35.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.901+0000 7f6b04a1d640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 0x7f6ae80a4e50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6b000646c0 tx=0x7f6b00067540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:35.903 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:35.902+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6af001f020 con 0x7f6af8099910 2024-03-28T18:50:36.319 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.318+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6af8002300 con 0x7f6af8099910 2024-03-28T18:50:36.321 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.320+0000 7f6afdffb640 1 -- 172.21.15.145:0/2314501623 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11482 (secure 0 0 0) 0x7f6af009b900 con 0x7f6af8099910 2024-03-28T18:50:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:50:36.321 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1607345926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1607345926},{"type":"v1","addr":"172.21.15.145:6835","nonce":1607345926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":25603,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/2973467272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":2973467272},{"type":"v1","addr":"172.21.15.192:6835","nonce":2973467272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:35:50.266038+0000","modified":"2024-03-28T18:41:47.139048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24496,"mds_1":24491,"mds_2":24469,"mds_3":24461,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14556":{"gid":14556,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6837/3754178140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3754178140},{"type":"v1","addr":"172.21.15.145:6837","nonce":3754178140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6839/1703430127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6838","nonce":1703430127},{"type":"v1","addr":"172.21.15.145:6839","nonce":1703430127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":9,"addr":"172.21.15.145:6841/2292179731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6840","nonce":2292179731},{"type":"v1","addr":"172.21.15.145:6841","nonce":2292179731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24395":{"gid":24395,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/3835667312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3835667312},{"type":"v1","addr":"172.21.15.164:6833","nonce":3835667312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/3704007459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":3704007459},{"type":"v1","addr":"172.21.15.192:6833","nonce":3704007459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/3793155920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":3793155920},{"type":"v1","addr":"172.21.15.164:6835","nonce":3793155920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24461":{"gid":24461,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6837/1634173775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":1634173775},{"type":"v1","addr":"172.21.15.164:6837","nonce":1634173775}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24469":{"gid":24469,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6837/2153191464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2153191464},{"type":"v1","addr":"172.21.15.192:6837","nonce":2153191464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24491":{"gid":24491,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.164:6839/644460294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":644460294},{"type":"v1","addr":"172.21.15.164:6839","nonce":644460294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.192:6839/2516430938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":2516430938},{"type":"v1","addr":"172.21.15.192:6839","nonce":2516430938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24461,24496,14631,24491,24469]},"id":1}]} 2024-03-28T18:50:36.322 INFO:teuthology.orchestra.run.smithi145.stderr:dumped fsmap epoch 31 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.323+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 msgr2=0x7f6ae80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.323+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 0x7f6ae80a4e50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6b000646c0 tx=0x7f6b00067540 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.323+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 msgr2=0x7f6af80ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.323+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6af000b3f0 tx=0x7f6af000b8b0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 shutdown_connections 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6af8099910 0x7f6af80ada50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6af8096f60 0x7f6af8143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f6ae80a2990 0x7f6ae80a4e50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 --2- 172.21.15.145:0/2314501623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6af80945f0 0x7f6af81435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 >> 172.21.15.145:0/2314501623 conn(0x7f6af80901d0 msgr2=0x7f6af8097920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:36.325 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 shutdown_connections 2024-03-28T18:50:36.326 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.324+0000 7f6b05a1f640 1 -- 172.21.15.145:0/2314501623 wait complete. 2024-03-28T18:50:36.344 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:50:36.406 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:36 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/1263309894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:36.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:36 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/1263309894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.520+0000 7fbfa16ff640 1 -- 172.21.15.145:0/612964306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1032a0 msgr2=0x7fbf9c109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.520+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/612964306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c109b30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009ca0 tx=0x7fbf9402c120 comp rx=0 tx=0).stop 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.520+0000 7fbfa16ff640 1 -- 172.21.15.145:0/612964306 shutdown_connections 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/612964306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c109b30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/612964306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/612964306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf9c1016f0 0x7fbf9c101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 -- 172.21.15.145:0/612964306 >> 172.21.15.145:0/612964306 conn(0x7fbf9c0fcc40 msgr2=0x7fbf9c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 -- 172.21.15.145:0/612964306 shutdown_connections 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.521+0000 7fbfa16ff640 1 -- 172.21.15.145:0/612964306 wait complete. 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.522+0000 7fbfa16ff640 1 Processor -- start 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.522+0000 7fbfa16ff640 1 -- start start 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.522+0000 7fbfa16ff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf9c1016f0 0x7fbf9c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.522+0000 7fbfa16ff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbfa16ff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbfa16ff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c10d3e0 con 0x7fbf9c1016f0 2024-03-28T18:50:36.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:54792/0 (socket says 172.21.15.145:54792) 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9a7fc640 1 -- 172.21.15.145:0/2948777913 learned_addr learned my addr 172.21.15.145:0/2948777913 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbfa16ff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c10d260 con 0x7fbf9c1028e0 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c10d560 con 0x7fbf9c1032a0 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 -- 172.21.15.145:0/2948777913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 msgr2=0x7fbf9c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 -- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf9c1016f0 msgr2=0x7fbf9c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf9c1016f0 0x7fbf9c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.523+0000 7fbf9b7fe640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c19f6b0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.524+0000 7fbf9b7fe640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c19f040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf94037040 tx=0x7fbf94035cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:36.526 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:36 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/1263309894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:36.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.525+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94039040 con 0x7fbf9c1032a0 2024-03-28T18:50:36.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.525+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf94003e50 con 0x7fbf9c1032a0 2024-03-28T18:50:36.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.525+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9403d5f0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.525+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c0738d0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.526+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c073d60 con 0x7fbf9c1032a0 2024-03-28T18:50:36.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.527+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c108a10 con 0x7fbf9c1032a0 2024-03-28T18:50:36.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.529+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbf94004040 con 0x7fbf9c1032a0 2024-03-28T18:50:36.531 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.530+0000 7fbf83fff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 0x7fbf840a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:36.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.530+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbf940f0bb0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.531+0000 7fbf9affd640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 0x7fbf840a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:36.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.532+0000 7fbf9affd640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 0x7fbf840a4ec0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c102740 tx=0x7fbf90006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:36.538 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.537+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbf940bd8c0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.860 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.859+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf9c02d1f0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.862 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.861+0000 7fbf83fff640 1 -- 172.21.15.145:0/2948777913 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24843 (secure 0 0 0) 0x7fbf940bd1c0 con 0x7fbf9c1032a0 2024-03-28T18:50:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-28T18:50:36.863 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":95,"fsid":"b991dbde-ed30-11ee-95d0-87774f69a715","created":"2024-03-28T18:28:01.180162+0000","modified":"2024-03-28T18:37:54.633430+0000","last_up_change":"2024-03-28T18:34:59.751844+0000","last_in_change":"2024-03-28T18:34:40.801224+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:31:13.733117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:35:48.277653+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:35:49.458851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T18:35:53.237697+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d20b632f-9b4a-4f94-82e3-3b5668acd7f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6803","nonce":2900391364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6805","nonce":2900391364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6809","nonce":2900391364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2900391364},{"type":"v1","addr":"172.21.15.145:6807","nonce":2900391364}]},"public_addr":"172.21.15.145:6803/2900391364","cluster_addr":"172.21.15.145:6805/2900391364","heartbeat_back_addr":"172.21.15.145:6809/2900391364","heartbeat_front_addr":"172.21.15.145:6807/2900391364","state":["exists","up"]},{"osd":1,"uuid":"70cfca66-0490-4c8f-9d54-0a64f05392a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6801","nonce":3569377863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6803","nonce":3569377863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6807","nonce":3569377863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3569377863},{"type":"v1","addr":"172.21.15.164:6805","nonce":3569377863}]},"public_addr":"172.21.15.164:6801/3569377863","cluster_addr":"172.21.15.164:6803/3569377863","heartbeat_back_addr":"172.21.15.164:6807/3569377863","heartbeat_front_addr":"172.21.15.164:6805/3569377863","state":["exists","up"]},{"osd":2,"uuid":"36e9e51d-79a5-402a-bf8b-9bb427e8e847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6801","nonce":629223859}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6803","nonce":629223859}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6807","nonce":629223859}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":629223859},{"type":"v1","addr":"172.21.15.192:6805","nonce":629223859}]},"public_addr":"172.21.15.192:6801/629223859","cluster_addr":"172.21.15.192:6803/629223859","heartbeat_back_addr":"172.21.15.192:6807/629223859","heartbeat_front_addr":"172.21.15.192:6805/629223859","state":["exists","up"]},{"osd":3,"uuid":"752a99f2-3b2f-4964-8e7c-1fd521d16fff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6811","nonce":1910028611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6813","nonce":1910028611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6817","nonce":1910028611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1910028611},{"type":"v1","addr":"172.21.15.145:6815","nonce":1910028611}]},"public_addr":"172.21.15.145:6811/1910028611","cluster_addr":"172.21.15.145:6813/1910028611","heartbeat_back_addr":"172.21.15.145:6817/1910028611","heartbeat_front_addr":"172.21.15.145:6815/1910028611","state":["exists","up"]},{"osd":4,"uuid":"fea7e9f1-4577-439a-81a8-5b6bf2e810e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6809","nonce":2404350787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6811","nonce":2404350787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6815","nonce":2404350787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2404350787},{"type":"v1","addr":"172.21.15.164:6813","nonce":2404350787}]},"public_addr":"172.21.15.164:6809/2404350787","cluster_addr":"172.21.15.164:6811/2404350787","heartbeat_back_addr":"172.21.15.164:6815/2404350787","heartbeat_front_addr":"172.21.15.164:6813/2404350787","state":["exists","up"]},{"osd":5,"uuid":"f883f88f-77ee-4a4e-9d4e-b7f0469c50fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6809","nonce":22255675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6811","nonce":22255675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6815","nonce":22255675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":22255675},{"type":"v1","addr":"172.21.15.192:6813","nonce":22255675}]},"public_addr":"172.21.15.192:6809/22255675","cluster_addr":"172.21.15.192:6811/22255675","heartbeat_back_addr":"172.21.15.192:6815/22255675","heartbeat_front_addr":"172.21.15.192:6813/22255675","state":["exists","up"]},{"osd":6,"uuid":"bf2bf55a-48f0-46b4-acbd-f8c012e9b39d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6819","nonce":2385620152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6821","nonce":2385620152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6825","nonce":2385620152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2385620152},{"type":"v1","addr":"172.21.15.145:6823","nonce":2385620152}]},"public_addr":"172.21.15.145:6819/2385620152","cluster_addr":"172.21.15.145:6821/2385620152","heartbeat_back_addr":"172.21.15.145:6825/2385620152","heartbeat_front_addr":"172.21.15.145:6823/2385620152","state":["exists","up"]},{"osd":7,"uuid":"532710ac-62e7-479f-885d-5604af92fd37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6817","nonce":795549152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6819","nonce":795549152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6823","nonce":795549152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":795549152},{"type":"v1","addr":"172.21.15.164:6821","nonce":795549152}]},"public_addr":"172.21.15.164:6817/795549152","cluster_addr":"172.21.15.164:6819/795549152","heartbeat_back_addr":"172.21.15.164:6823/795549152","heartbeat_front_addr":"172.21.15.164:6821/795549152","state":["exists","up"]},{"osd":8,"uuid":"dc3ae89b-0840-440c-8bce-854d680e9538","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6817","nonce":1092086458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6819","nonce":1092086458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6823","nonce":1092086458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1092086458},{"type":"v1","addr":"172.21.15.192:6821","nonce":1092086458}]},"public_addr":"172.21.15.192:6817/1092086458","cluster_addr":"172.21.15.192:6819/1092086458","heartbeat_back_addr":"172.21.15.192:6823/1092086458","heartbeat_front_addr":"172.21.15.192:6821/1092086458","state":["exists","up"]},{"osd":9,"uuid":"5c299158-d952-43b7-8ef1-6da9ab558f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6827","nonce":2302293093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6829","nonce":2302293093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6833","nonce":2302293093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":2302293093},{"type":"v1","addr":"172.21.15.145:6831","nonce":2302293093}]},"public_addr":"172.21.15.145:6827/2302293093","cluster_addr":"172.21.15.145:6829/2302293093","heartbeat_back_addr":"172.21.15.145:6833/2302293093","heartbeat_front_addr":"172.21.15.145:6831/2302293093","state":["exists","up"]},{"osd":10,"uuid":"bce4fe04-fb9d-4108-ab8f-420cae8243ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6825","nonce":3964117002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6827","nonce":3964117002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6831","nonce":3964117002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3964117002},{"type":"v1","addr":"172.21.15.164:6829","nonce":3964117002}]},"public_addr":"172.21.15.164:6825/3964117002","cluster_addr":"172.21.15.164:6827/3964117002","heartbeat_back_addr":"172.21.15.164:6831/3964117002","heartbeat_front_addr":"172.21.15.164:6829/3964117002","state":["exists","up"]},{"osd":11,"uuid":"d95d0633-327c-4fe9-a150-07eaa70cbcec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6825","nonce":3207430647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6827","nonce":3207430647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6831","nonce":3207430647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3207430647},{"type":"v1","addr":"172.21.15.192:6829","nonce":3207430647}]},"public_addr":"172.21.15.192:6825/3207430647","cluster_addr":"172.21.15.192:6827/3207430647","heartbeat_back_addr":"172.21.15.192:6831/3207430647","heartbeat_front_addr":"172.21.15.192:6829/3207430647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:23.218972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:30:47.036298+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:10.735047+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:31:35.942258+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:00.725710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:25.579440+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:32:50.726811+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:16.060863+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:33:40.938559+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:07.390076+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:33.400725+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:34:58.564423+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.11","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:0/606368710":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/4006486926":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/2249018729":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3480949122":"2024-03-29T18:28:28.346767+0000","172.21.15.145:0/832150839":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6800/2044834522":"2024-03-29T18:28:28.346767+0000","172.21.15.145:6801/2089479814":"2024-03-29T18:28:53.394017+0000","172.21.15.145:0/3638297666":"2024-03-29T18:28:53.394017+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 msgr2=0x7fbf840a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 0x7fbf840a4ec0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c102740 tx=0x7fbf90006c90 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 msgr2=0x7fbf9c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c19f040 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbf94037040 tx=0x7fbf94035cf0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 shutdown_connections 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fbf840a2a00 0x7fbf840a4ec0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf9c1032a0 0x7fbf9c19f040 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf9c1028e0 0x7fbf9c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 --2- 172.21.15.145:0/2948777913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf9c1016f0 0x7fbf9c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 >> 172.21.15.145:0/2948777913 conn(0x7fbf9c0fcc40 msgr2=0x7fbf9c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 shutdown_connections 2024-03-28T18:50:36.867 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:36.865+0000 7fbfa16ff640 1 -- 172.21.15.145:0/2948777913 wait complete. 2024-03-28T18:50:36.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T18:50:37.068 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.066+0000 7f44fae6c640 1 -- 172.21.15.145:0/1684482690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 msgr2=0x7f44f4102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.068 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.066+0000 7f44fae6c640 1 --2- 172.21.15.145:0/1684482690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 0x7f44f4102360 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009ca0 tx=0x7f44ec02c120 comp rx=0 tx=0).stop 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 -- 172.21.15.145:0/1684482690 shutdown_connections 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 --2- 172.21.15.145:0/1684482690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44f4105250 0x7f44f41056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 --2- 172.21.15.145:0/1684482690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f41028a0 0x7f44f4104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 --2- 172.21.15.145:0/1684482690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 0x7f44f4102360 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 -- 172.21.15.145:0/1684482690 >> 172.21.15.145:0/1684482690 conn(0x7f44f40fbb50 msgr2=0x7f44f40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 -- 172.21.15.145:0/1684482690 shutdown_connections 2024-03-28T18:50:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.068+0000 7f44fae6c640 1 -- 172.21.15.145:0/1684482690 wait complete. 2024-03-28T18:50:37.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 Processor -- start 2024-03-28T18:50:37.070 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 -- start start 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 0x7f44f419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44f41028a0 0x7f44f419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4113a70 con 0x7f44f40fff30 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f41138f0 con 0x7f44f4105250 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.069+0000 7f44fae6c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4113bf0 con 0x7f44f41028a0 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.145:54814/0 (socket says 172.21.15.145:54814) 2024-03-28T18:50:37.071 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 -- 172.21.15.145:0/2573105530 learned_addr learned my addr 172.21.15.145:0/2573105530 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 -- 172.21.15.145:0/2573105530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44f41028a0 msgr2=0x7f44f419e6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44f41028a0 0x7f44f419e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 -- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 msgr2=0x7f44f419e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 0x7f44f419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 -- 172.21.15.145:0/2573105530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f4106680 con 0x7f44f4105250 2024-03-28T18:50:37.072 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.070+0000 7f44f9409640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b3f0 tx=0x7f44e800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:37.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.074+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8015020 con 0x7f44f4105250 2024-03-28T18:50:37.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.074+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44e8011d10 con 0x7f44f4105250 2024-03-28T18:50:37.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.075+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800f2e0 con 0x7f44f4105250 2024-03-28T18:50:37.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.075+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f4106970 con 0x7f44f4105250 2024-03-28T18:50:37.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.075+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f4106ef0 con 0x7f44f4105250 2024-03-28T18:50:37.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.077+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f40fd640 con 0x7f44f4105250 2024-03-28T18:50:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.084+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f44e8018020 con 0x7f44f4105250 2024-03-28T18:50:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.084+0000 7f44f1ffb640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 0x7f44dc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.085+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f44e80cec80 con 0x7f44f4105250 2024-03-28T18:50:37.086 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.085+0000 7f44f8c08640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 0x7f44dc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.087 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.086+0000 7f44f8c08640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 0x7f44dc0a4ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f44ec004af0 tx=0x7f44ec005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:37.088 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.087+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44e801f020 con 0x7f44f4105250 2024-03-28T18:50:37.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-mon[36589]: pgmap v736: 129 pgs: 129 active+clean; 5.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 167 MiB/s wr, 1.05k op/s 2024-03-28T18:50:37.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2314501623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2948777913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mon[36248]: pgmap v736: 129 pgs: 129 active+clean; 5.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 167 MiB/s wr, 1.05k op/s 2024-03-28T18:50:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2314501623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2948777913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.503 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.502+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f44f40622a0 con 0x7f44f4105250 2024-03-28T18:50:37.507 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mon[32151]: pgmap v736: 129 pgs: 129 active+clean; 5.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 167 MiB/s wr, 1.05k op/s 2024-03-28T18:50:37.507 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2314501623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.508 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2948777913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:50:37.520 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.519+0000 7f44f1ffb640 1 -- 172.21.15.145:0/2573105530 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f44e809b990 con 0x7f44f4105250 2024-03-28T18:50:37.520 INFO:teuthology.orchestra.run.smithi145.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 msgr2=0x7f44dc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 0x7f44dc0a4ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f44ec004af0 tx=0x7f44ec005ef0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 msgr2=0x7f44f41a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b3f0 tx=0x7f44e800b8b0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 shutdown_connections 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f4105250 0x7f44f41a5750 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f44dc0a2a00 0x7f44dc0a4ec0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44f41028a0 0x7f44f419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 --2- 172.21.15.145:0/2573105530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44f40fff30 0x7f44f419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 >> 172.21.15.145:0/2573105530 conn(0x7f44f40fbb50 msgr2=0x7f44f4103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:37.524 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.522+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 shutdown_connections 2024-03-28T18:50:37.525 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.523+0000 7f44fae6c640 1 -- 172.21.15.145:0/2573105530 wait complete. 2024-03-28T18:50:37.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T18:50:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 -- 172.21.15.145:0/2464217369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 msgr2=0x7f7c20102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/2464217369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20102d40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009860 tx=0x7f7c1002bb70 comp rx=0 tx=0).stop 2024-03-28T18:50:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 -- 172.21.15.145:0/2464217369 shutdown_connections 2024-03-28T18:50:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/2464217369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c20103280 0x7f7c20109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/2464217369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20102d40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/2464217369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c201016d0 0x7f7c20101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 -- 172.21.15.145:0/2464217369 >> 172.21.15.145:0/2464217369 conn(0x7f7c200fcc40 msgr2=0x7f7c200ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 -- 172.21.15.145:0/2464217369 shutdown_connections 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.671+0000 7f7c27ac0640 1 -- 172.21.15.145:0/2464217369 wait complete. 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 Processor -- start 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 -- start start 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 0x7f7c20197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20103280 0x7f7c2019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010d350 con 0x7f7c201016d0 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010d1d0 con 0x7f7c201028c0 2024-03-28T18:50:37.675 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c27ac0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010d4d0 con 0x7f7c20103280 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2585c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 0x7f7c20197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2505b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2585c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 0x7f7c20197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41868/0 (socket says 172.21.15.145:41868) 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2585c640 1 -- 172.21.15.145:0/3854789245 learned_addr learned my addr 172.21.15.145:0/3854789245 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2605d640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20103280 0x7f7c2019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2505b640 1 -- 172.21.15.145:0/3854789245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20103280 msgr2=0x7f7c2019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2505b640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20103280 0x7f7c2019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2505b640 1 -- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 msgr2=0x7f7c20197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.672+0000 7f7c2505b640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 0x7f7c20197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c2505b640 1 -- 172.21.15.145:0/3854789245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2019f640 con 0x7f7c201028c0 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c2505b640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20197f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7c1002c0b0 tx=0x7f7c10032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:37.676 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10041020 con 0x7f7c201028c0 2024-03-28T18:50:37.677 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c1002c9f0 con 0x7f7c201028c0 2024-03-28T18:50:37.677 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10039040 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2019f8d0 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.673+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2019fd40 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.674+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c20108a70 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.675+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7c10044020 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.676+0000 7f7c16ffd640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 0x7f7c0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.676+0000 7f7c2585c640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 0x7f7c0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.676+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9609+0+0 (secure 0 0 0) 0x7f7c100f0410 con 0x7f7c201028c0 2024-03-28T18:50:37.678 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.676+0000 7f7c2585c640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 0x7f7c0c0a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0058c0 tx=0x7f7c1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:37.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.680+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c1004b020 con 0x7f7c201028c0 2024-03-28T18:50:37.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a[32147]: 2024-03-28T18:50:37.505+0000 7f3e1e7c8700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mds[70677]: -- 172.21.15.145:0/3534907284 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0c00aba40 con 0x55b0c007e800 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: starting mds.g at 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mds[71028]: -- 172.21.15.145:0/1261249338 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bba068fa40 con 0x55bba0662800 2024-03-28T18:50:37.777 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: starting mds.a at 2024-03-28T18:50:37.778 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.778 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.778 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mds[70069]: -- 172.21.15.145:0/2013134976 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561c179e64e0 con 0x561c179bc800 2024-03-28T18:50:37.778 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-mds[70069]: -- 172.21.15.145:0/2013134976 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c179e9a40 con 0x561c179bc800 2024-03-28T18:50:37.778 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:50:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: starting mds.d at 2024-03-28T18:50:37.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:37.898+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7c2002d200 con 0x7f7c201028c0 2024-03-28T18:50:37.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-mds[66879]: -- 172.21.15.192:0/898591141 <== mon.0 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d458265a40 con 0x55d458238800 2024-03-28T18:50:37.907 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: starting mds.i at 2024-03-28T18:50:37.907 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: starting mds.l at 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: 2024-03-28T18:50:37.613+0000 7f0ae111d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-mds[66563]: -- 172.21.15.192:0/1497684582 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c65c94ba40 con 0x55c65c91ec00 2024-03-28T18:50:37.908 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:50:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: starting mds.c at 2024-03-28T18:50:37.962 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.962 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.962 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mds[66813]: -- 172.21.15.164:0/2301889567 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558113a27a40 con 0x5581139fac00 2024-03-28T18:50:37.962 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: starting mds.b at 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mds[66497]: -- 172.21.15.164:0/200206217 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55855f911a40 con 0x55855f8e5000 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: starting mds.h at 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.963 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: starting mds.k at 2024-03-28T18:50:37.964 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: ignoring --setuser ceph since I am not root 2024-03-28T18:50:37.964 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: ignoring --setgroup ceph since I am not root 2024-03-28T18:50:37.964 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-mds[66185]: -- 172.21.15.164:0/3509147909 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dddf0d3a40 con 0x55dddf0a7000 2024-03-28T18:50:37.964 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:50:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: starting mds.e at 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2573105530' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:50:38.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/3854789245' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:38 smithi192 ceph-mon[36589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2573105530' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:50:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:50:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:50:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:50:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:50:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/3854789245' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:38 smithi164 ceph-mon[36248]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.523 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.522+0000 7f7c16ffd640 1 -- 172.21.15.145:0/3854789245 <== mon.2 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c10032e50 con 0x7f7c201028c0 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2573105530' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/3854789245' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.527 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:38 smithi145 ceph-mon[32151]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 msgr2=0x7f7c0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 0x7f7c0c0a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0058c0 tx=0x7f7c1c005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 msgr2=0x7f7c20197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20197f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7c1002c0b0 tx=0x7f7c10032430 comp rx=0 tx=0).stop 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 shutdown_connections 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f7c0c0a2800 0x7f7c0c0a4cc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20103280 0x7f7c2019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c201028c0 0x7f7c20197f60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 --2- 172.21.15.145:0/3854789245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c201016d0 0x7f7c20197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 >> 172.21.15.145:0/3854789245 conn(0x7f7c200fcc40 msgr2=0x7f7c20105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:38.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 shutdown_connections 2024-03-28T18:50:38.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.526+0000 7f7c27ac0640 1 -- 172.21.15.145:0/3854789245 wait complete. 2024-03-28T18:50:38.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T18:50:38.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 -- 172.21.15.145:0/75097726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 msgr2=0x7fba481009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:38.657 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/75097726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba481009c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fba38009ca0 tx=0x7fba3802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 -- 172.21.15.145:0/75097726 shutdown_connections 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/75097726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48102150 0x7fba481089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/75097726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48101790 0x7fba48101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/75097726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba481009c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 -- 172.21.15.145:0/75097726 >> 172.21.15.145:0/75097726 conn(0x7fba480fbb10 msgr2=0x7fba480fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.653+0000 7fba4cbd4640 1 -- 172.21.15.145:0/75097726 shutdown_connections 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 -- 172.21.15.145:0/75097726 wait complete. 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 Processor -- start 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 -- start start 2024-03-28T18:50:38.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48101790 0x7fba48197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48102150 0x7fba4819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4810c230 con 0x7fba481005a0 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4810c0b0 con 0x7fba48101790 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4cbd4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4810c3b0 con 0x7fba48102150 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.654+0000 7fba4659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41884/0 (socket says 172.21.15.145:41884) 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 -- 172.21.15.145:0/4019809816 learned_addr learned my addr 172.21.15.145:0/4019809816 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba46d9d640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48102150 0x7fba4819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba45d9b640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48101790 0x7fba48197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:38.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 -- 172.21.15.145:0/4019809816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48102150 msgr2=0x7fba4819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48102150 0x7fba4819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 -- 172.21.15.145:0/4019809816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48101790 msgr2=0x7fba48197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48101790 0x7fba48197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 -- 172.21.15.145:0/4019809816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4819f590 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4659c640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fba380324c0 tx=0x7fba38032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38041020 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba38034cd0 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4819f820 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38039040 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.655+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4819fd10 con 0x7fba481005a0 2024-03-28T18:50:38.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.656+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4819ff70 con 0x7fba481005a0 2024-03-28T18:50:38.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.657+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fba38044020 con 0x7fba481005a0 2024-03-28T18:50:38.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.658+0000 7fba377fe640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 0x7fba300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:38.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.658+0000 7fba45d9b640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 0x7fba300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:38.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.658+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9609+0+0 (secure 0 0 0) 0x7fba380f12b0 con 0x7fba481005a0 2024-03-28T18:50:38.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.658+0000 7fba45d9b640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 0x7fba300a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba48198e80 tx=0x7fba3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:38.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.661+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba3804b020 con 0x7fba481005a0 2024-03-28T18:50:38.842 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:38.840+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba481a0640 con 0x7fba481005a0 2024-03-28T18:50:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: pgmap v738: 129 pgs: 129 active+clean; 5.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 MiB/s wr, 1.45k op/s 2024-03-28T18:50:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:50:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:50:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: Cluster is now healthy 2024-03-28T18:50:39.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6833/1177161029,v1:172.21.15.192:6837/1177161029] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6833/2034156317,v1:172.21.15.164:6835/2034156317] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6838/3853796486,v1:172.21.15.192:6839/3853796486] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.192:6832/1849632418,v1:172.21.15.192:6836/1849632418] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6832/1280631890,v1:172.21.15.164:6834/1280631890] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6836/1124120733,v1:172.21.15.164:6837/1124120733] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.164:6838/2406759843,v1:172.21.15.164:6839/2406759843] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6836/3699594401,v1:172.21.15.145:6838/3699594401] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6840/1304197891,v1:172.21.15.145:6841/1304197891] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: mds.? [v2:172.21.15.145:6837/3671821286,v1:172.21.15.145:6839/3671821286] up:boot 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: fsmap 12 up:standby 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:50:39.408 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:39 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:39.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: pgmap v738: 129 pgs: 129 active+clean; 5.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 MiB/s wr, 1.45k op/s 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: Cluster is now healthy 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6833/1177161029,v1:172.21.15.192:6837/1177161029] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6833/2034156317,v1:172.21.15.164:6835/2034156317] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6838/3853796486,v1:172.21.15.192:6839/3853796486] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.192:6832/1849632418,v1:172.21.15.192:6836/1849632418] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6832/1280631890,v1:172.21.15.164:6834/1280631890] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6836/1124120733,v1:172.21.15.164:6837/1124120733] up:boot 2024-03-28T18:50:39.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.164:6838/2406759843,v1:172.21.15.164:6839/2406759843] up:boot 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6836/3699594401,v1:172.21.15.145:6838/3699594401] up:boot 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6840/1304197891,v1:172.21.15.145:6841/1304197891] up:boot 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: mds.? [v2:172.21.15.145:6837/3671821286,v1:172.21.15.145:6839/3671821286] up:boot 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: fsmap 12 up:standby 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:50:39.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:50:39.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:50:39.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:39 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:39.522 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.521+0000 7fba377fe640 1 -- 172.21.15.145:0/4019809816 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v97) v1 ==== 181+0+0 (secure 0 0 0) 0x7fba380bde80 con 0x7fba481005a0 2024-03-28T18:50:39.522 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_metadata' removed 2024-03-28T18:50:39.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.524+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 msgr2=0x7fba300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:39.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.524+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 0x7fba300a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba48198e80 tx=0x7fba3c005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:39.526 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.524+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 msgr2=0x7fba48197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.524+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fba380324c0 tx=0x7fba38032f40 comp rx=0 tx=0).stop 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 shutdown_connections 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fba300a2800 0x7fba300a4cc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba48102150 0x7fba4819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba48101790 0x7fba48197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 --2- 172.21.15.145:0/4019809816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba481005a0 0x7fba48197970 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 >> 172.21.15.145:0/4019809816 conn(0x7fba480fbb10 msgr2=0x7fba480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 shutdown_connections 2024-03-28T18:50:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.525+0000 7fba4cbd4640 1 -- 172.21.15.145:0/4019809816 wait complete. 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: pgmap v738: 129 pgs: 129 active+clean; 5.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 204 MiB/s wr, 1.45k op/s 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: Cluster is now healthy 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6833/1177161029,v1:172.21.15.192:6837/1177161029] up:boot 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6833/2034156317,v1:172.21.15.164:6835/2034156317] up:boot 2024-03-28T18:50:39.528 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6838/3853796486,v1:172.21.15.192:6839/3853796486] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.192:6832/1849632418,v1:172.21.15.192:6836/1849632418] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6832/1280631890,v1:172.21.15.164:6834/1280631890] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6836/1124120733,v1:172.21.15.164:6837/1124120733] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.164:6838/2406759843,v1:172.21.15.164:6839/2406759843] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6836/3699594401,v1:172.21.15.145:6838/3699594401] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6840/1304197891,v1:172.21.15.145:6841/1304197891] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: mds.? [v2:172.21.15.145:6837/3671821286,v1:172.21.15.145:6839/3671821286] up:boot 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: fsmap 12 up:standby 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:50:39.529 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='mgr.14152 172.21.15.145:0/1184906339' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:50:39.530 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:39 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:39.543 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T18:50:39.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/2759235048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01016b0 msgr2=0x7f3ce0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:39.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/2759235048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01016b0 0x7f3ce0101ad0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009900 tx=0x7f3cdc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:39.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/2759235048 shutdown_connections 2024-03-28T18:50:39.658 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/2759235048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 0x7f3ce0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/2759235048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01028a0 0x7f3ce0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/2759235048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01016b0 0x7f3ce0101ad0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/2759235048 >> 172.21.15.145:0/2759235048 conn(0x7f3ce00fcc40 msgr2=0x7f3ce00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/2759235048 shutdown_connections 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/2759235048 wait complete. 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 Processor -- start 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.654+0000 7f3ce7f9d640 1 -- start start 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 0x7f3ce0197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:39.659 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 0x7f3ce019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce010d300 con 0x7f3ce01028a0 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce010d180 con 0x7f3ce01016b0 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce010d480 con 0x7f3ce0103260 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41898/0 (socket says 172.21.15.145:41898) 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 -- 172.21.15.145:0/4031671773 learned_addr learned my addr 172.21.15.145:0/4031671773 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce653a640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 0x7f3ce019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:39.660 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5d39640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 0x7f3ce0197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 -- 172.21.15.145:0/4031671773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 msgr2=0x7f3ce019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 0x7f3ce019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 -- 172.21.15.145:0/4031671773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 msgr2=0x7f3ce0197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 0x7f3ce0197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 -- 172.21.15.145:0/4031671773 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce019f640 con 0x7f3ce01028a0 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5d39640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 0x7f3ce0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce5538640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000d950 tx=0x7f3cd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0017070 con 0x7f3ce01028a0 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd0009c30 con 0x7f3ce01028a0 2024-03-28T18:50:39.661 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd000f6b0 con 0x7f3ce01028a0 2024-03-28T18:50:39.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce019f930 con 0x7f3ce01028a0 2024-03-28T18:50:39.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.655+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce019fe80 con 0x7f3ce01028a0 2024-03-28T18:50:39.662 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.656+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce0108af0 con 0x7f3ce01028a0 2024-03-28T18:50:39.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.657+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3cd00109d0 con 0x7f3ce01028a0 2024-03-28T18:50:39.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.658+0000 7f3cd6ffd640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 0x7f3ccc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:39.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.658+0000 7f3ce5d39640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 0x7f3ccc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:39.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.658+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9055+0+0 (secure 0 0 0) 0x7f3cd00cea40 con 0x7f3ce01028a0 2024-03-28T18:50:39.663 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.658+0000 7f3ce5d39640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 0x7f3ccc0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc02c7b0 tx=0x7f3cdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:39.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.662+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3cd0009de0 con 0x7f3ce01028a0 2024-03-28T18:50:39.841 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:39.840+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ce002d070 con 0x7f3ce01028a0 2024-03-28T18:50:40.528 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.526+0000 7f3cd6ffd640 1 -- 172.21.15.145:0/4031671773 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v98) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3cd009b7b0 con 0x7f3ce01028a0 2024-03-28T18:50:40.528 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_data' removed 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.530+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 msgr2=0x7f3ccc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.530+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 0x7f3ccc0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc02c7b0 tx=0x7f3cdc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.530+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 msgr2=0x7f3ce0197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.530+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000d950 tx=0x7f3cd000de10 comp rx=0 tx=0).stop 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 shutdown_connections 2024-03-28T18:50:40.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7f3ccc0a2800 0x7f3ccc0a4cc0 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc02c7b0 tx=0x7f3cdc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3ce0103260 0x7f3ce019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ce01028a0 0x7f3ce0197f60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 --2- 172.21.15.145:0/4031671773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ce01016b0 0x7f3ce0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 >> 172.21.15.145:0/4031671773 conn(0x7f3ce00fcc40 msgr2=0x7f3ce0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 shutdown_connections 2024-03-28T18:50:40.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.531+0000 7f3ce7f9d640 1 -- 172.21.15.145:0/4031671773 wait complete. 2024-03-28T18:50:40.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.660+0000 7fb12a8c6640 1 -- 172.21.15.145:0/3328868775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124102890 msgr2=0x7fb124102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.660+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/3328868775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124102890 0x7fb124102d10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb118009900 tx=0x7fb11802bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.660+0000 7fb12a8c6640 1 -- 172.21.15.145:0/3328868775 shutdown_connections 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/3328868775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb124103250 0x7fb124109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/3328868775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124102890 0x7fb124102d10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/3328868775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb1241016a0 0x7fb124101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- 172.21.15.145:0/3328868775 >> 172.21.15.145:0/3328868775 conn(0x7fb1240fcc30 msgr2=0x7fb1240ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- 172.21.15.145:0/3328868775 shutdown_connections 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- 172.21.15.145:0/3328868775 wait complete. 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 Processor -- start 2024-03-28T18:50:40.664 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- start start 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 0x7fb124197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb124102890 0x7fb124197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12410d320 con 0x7fb124103250 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12410d1a0 con 0x7fb1241016a0 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.661+0000 7fb12a8c6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12410d4a0 con 0x7fb124102890 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:41916/0 (socket says 172.21.15.145:41916) 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 -- 172.21.15.145:0/2125199896 learned_addr learned my addr 172.21.15.145:0/2125199896 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb1237fe640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb124102890 0x7fb124197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb123fff640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 0x7fb124197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 -- 172.21.15.145:0/2125199896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb124102890 msgr2=0x7fb124197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb124102890 0x7fb124197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.665 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 -- 172.21.15.145:0/2125199896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 msgr2=0x7fb124197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 0x7fb124197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 -- 172.21.15.145:0/2125199896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12419f620 con 0x7fb124103250 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb123fff640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 0x7fb124197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb128e63640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00cad0 tx=0x7fb11c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00ae20 con 0x7fb124103250 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12419f910 con 0x7fb124103250 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb11c005020 con 0x7fb124103250 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00f6f0 con 0x7fb124103250 2024-03-28T18:50:40.666 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.662+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12419fde0 con 0x7fb124103250 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.663+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1241089e0 con 0x7fb124103250 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.664+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb11c0051d0 con 0x7fb124103250 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.665+0000 7fb1217fa640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 0x7fb10c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.665+0000 7fb123fff640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 0x7fb10c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.665+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8496+0+0 (secure 0 0 0) 0x7fb11c0cec00 con 0x7fb124103250 2024-03-28T18:50:40.667 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.665+0000 7fb123fff640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 0x7fb10c0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb1140058c0 tx=0x7fb114005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:40.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.669+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb11c014030 con 0x7fb124103250 2024-03-28T18:50:40.849 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:40.846+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb1240622a0 con 0x7fb124103250 2024-03-28T18:50:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:40 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:40 smithi192 ceph-mon[36589]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:50:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:40 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:40.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:40 smithi192 ceph-mon[36589]: pgmap v740: 97 pgs: 97 active+clean; 2.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 1.16k op/s 2024-03-28T18:50:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:40 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:40 smithi164 ceph-mon[36248]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:50:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:40 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:40.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:40 smithi164 ceph-mon[36248]: pgmap v740: 97 pgs: 97 active+clean; 2.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 1.16k op/s 2024-03-28T18:50:41.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:40 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4019809816' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:41.026 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:40 smithi145 ceph-mon[32151]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:50:41.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:40 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:41.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:40 smithi145 ceph-mon[32151]: pgmap v740: 97 pgs: 97 active+clean; 2.0 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 1.16k op/s 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.528+0000 7fb1217fa640 1 -- 172.21.15.145:0/2125199896 <== mon.0 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v99) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb11c09bba0 con 0x7fb124103250 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi145.stderr:pool 'cephfs_data_ec' removed 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.530+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 msgr2=0x7fb10c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.530+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 0x7fb10c0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb1140058c0 tx=0x7fb114005850 comp rx=0 tx=0).stop 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.530+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 msgr2=0x7fb12419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.530+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb11c00cad0 tx=0x7fb11c00cf90 comp rx=0 tx=0).stop 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 shutdown_connections 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb124103250 0x7fb12419efb0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.145:6800/82375532,v1:172.21.15.145:6801/82375532] conn(0x7fb10c0a2800 0x7fb10c0a4cc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb124102890 0x7fb124197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.532 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 --2- 172.21.15.145:0/2125199896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1241016a0 0x7fb124197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 >> 172.21.15.145:0/2125199896 conn(0x7fb1240fcc30 msgr2=0x7fb124105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 shutdown_connections 2024-03-28T18:50:41.533 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-28T18:50:41.531+0000 7fb12a8c6640 1 -- 172.21.15.145:0/2125199896 wait complete. 2024-03-28T18:50:41.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:50:41.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:50:41.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:50:41.662 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T18:50:41.662 DEBUG:teuthology.orchestra.run.smithi145:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:50:41.677 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:50:41.698 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:50:41.717 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T18:50:41.717 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T18:50:41.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a 2024-03-28T18:50:41.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:41 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:41.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:41 smithi192 ceph-mon[36589]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:50:41.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:41 smithi192 ceph-mon[36589]: from='client.? 172.21.15.145:0/2125199896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:41.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:41 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:41.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:41 smithi164 ceph-mon[36248]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:50:41.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:41 smithi164 ceph-mon[36248]: from='client.? 172.21.15.145:0/2125199896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:42.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:41 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/4031671773' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:50:42.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:41 smithi145 ceph-mon[32151]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:50:42.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:41 smithi145 ceph-mon[32151]: from='client.? 172.21.15.145:0/2125199896' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:50:42.027 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:41 smithi145 systemd[1]: Stopping Ceph mon.a for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:42.437 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a[32147]: 2024-03-28T18:50:42.109+0000 7f3e23fd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:42.437 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a[32147]: 2024-03-28T18:50:42.109+0000 7f3e23fd3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T18:50:42.437 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:42 smithi145 podman[90162]: 2024-03-28 18:50:42.15196248 +0000 UTC m=+0.242798740 container died 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T18:50:42.776 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:42 smithi145 podman[90162]: 2024-03-28 18:50:42.51925626 +0000 UTC m=+0.610092518 container cleanup 01c03d436a16787e2e89c17dc6526b270108923e48b96e6409d2110fab766c56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:50:42.777 INFO:journalctl@ceph.mon.a.smithi145.stdout:Mar 28 18:50:42 smithi145 bash[90162]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-a 2024-03-28T18:50:42.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T18:50:43.038 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:43.038 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T18:50:43.038 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T18:50:43.039 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.b 2024-03-28T18:50:43.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:43 smithi164 systemd[1]: Stopping Ceph mon.b for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-b[36244]: 2024-03-28T18:50:43.403+0000 7fa360da6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-b[36244]: 2024-03-28T18:50:43.403+0000 7fa360da6700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T18:50:43.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:43 smithi164 podman[71280]: 2024-03-28 18:50:43.558115313 +0000 UTC m=+0.282020223 container died a2f122f90cc3268b205ade63a08d50f2842cef9aea6da68c62a6384ca392e220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:50:44.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:44 smithi164 podman[71280]: 2024-03-28 18:50:44.055499078 +0000 UTC m=+0.779403971 container cleanup a2f122f90cc3268b205ade63a08d50f2842cef9aea6da68c62a6384ca392e220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:50:44.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Mar 28 18:50:44 smithi164 bash[71280]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-b 2024-03-28T18:50:44.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T18:50:44.670 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:44.671 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T18:50:44.671 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T18:50:44.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.c 2024-03-28T18:50:45.019 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:44 smithi192 systemd[1]: Stopping Ceph mon.c for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:45.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c[36585]: 2024-03-28T18:50:45.018+0000 7f37bfc05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:45.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c[36585]: 2024-03-28T18:50:45.018+0000 7f37bfc05700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T18:50:45.407 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:45 smithi192 podman[71645]: 2024-03-28 18:50:45.21222824 +0000 UTC m=+0.322564883 container died b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T18:50:45.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:45 smithi192 podman[71645]: 2024-03-28 18:50:45.543744423 +0000 UTC m=+0.654081056 container cleanup b85af841afcb2882245bbf05ef3c82a23b4f2c0b233889e890b7d06da3e162c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:50:45.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 28 18:50:45 smithi192 bash[71645]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mon-c 2024-03-28T18:50:46.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T18:50:46.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:46.060 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T18:50:46.061 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T18:50:46.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x 2024-03-28T18:50:46.381 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 systemd[1]: Stopping Ceph mgr.x for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:46.732 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:50:46.380+0000 7f0fbe4a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:46.733 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x[32380]: 2024-03-28T18:50:46.380+0000 7f0fbe4a3700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T18:50:46.733 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 podman[90281]: 2024-03-28 18:50:46.411414733 +0000 UTC m=+0.155752391 container died f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T18:50:47.027 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 podman[90281]: 2024-03-28 18:50:46.840514963 +0000 UTC m=+0.584852614 container cleanup f599c4e55841b964f0fe9da893af58ce16df374e8a5b09978d4ab322ad5e3427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T18:50:47.027 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Mar 28 18:50:46 smithi145 bash[90281]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-x 2024-03-28T18:50:47.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T18:50:47.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:47.259 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T18:50:47.259 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T18:50:47.259 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.y 2024-03-28T18:50:47.598 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:50:47 smithi164 systemd[1]: Stopping Ceph mgr.y for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:47.911 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:50:47 smithi164 podman[71411]: 2024-03-28 18:50:47.598040683 +0000 UTC m=+0.151677646 container died 251d73a10f39be18c6ddba40cde2f49892a5b1ef20bb70a4bb8c1b22a691f05b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T18:50:48.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:50:48 smithi164 podman[71411]: 2024-03-28 18:50:48.019342557 +0000 UTC m=+0.572979501 container cleanup 251d73a10f39be18c6ddba40cde2f49892a5b1ef20bb70a4bb8c1b22a691f05b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T18:50:48.212 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 28 18:50:48 smithi164 bash[71411]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-y 2024-03-28T18:50:48.483 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T18:50:48.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:48.522 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T18:50:48.522 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T18:50:48.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.z 2024-03-28T18:50:48.879 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:50:48 smithi192 systemd[1]: Stopping Ceph mgr.z for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:49.157 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:50:48 smithi192 podman[71767]: 2024-03-28 18:50:48.879449721 +0000 UTC m=+0.170843347 container died 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T18:50:49.561 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:50:49 smithi192 podman[71767]: 2024-03-28 18:50:49.310242021 +0000 UTC m=+0.601635640 container cleanup 3e659b1275d616876bf1a569c4c041f77698d7445c41a5372e7cda8cc9bea3fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T18:50:49.691 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 28 18:50:49 smithi192 bash[71767]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mgr-z 2024-03-28T18:50:49.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T18:50:49.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:49.809 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T18:50:49.810 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T18:50:49.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.0 2024-03-28T18:50:50.111 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:49 smithi145 systemd[1]: Stopping Ceph osd.0 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:50.527 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0[43962]: 2024-03-28T18:50:50.111+0000 7fe6b2747700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:50.527 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0[43962]: 2024-03-28T18:50:50.111+0000 7fe6b2747700 -1 osd.0 99 *** Got signal Terminated *** 2024-03-28T18:50:50.527 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0[43962]: 2024-03-28T18:50:50.111+0000 7fe6b2747700 -1 osd.0 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:50:55.962 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:55 smithi145 podman[90403]: 2024-03-28 18:50:55.684634515 +0000 UTC m=+5.687632248 container died e1140b63173769ed53c52282832764945f79761576c580607efbdca95a0ae6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T18:50:56.276 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90403]: 2024-03-28 18:50:56.053606499 +0000 UTC m=+6.056604232 container cleanup e1140b63173769ed53c52282832764945f79761576c580607efbdca95a0ae6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:50:56.277 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 bash[90403]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0 2024-03-28T18:50:56.588 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90420]: 2024-03-28 18:50:56.338575272 +0000 UTC m=+0.649253433 container remove e1140b63173769ed53c52282832764945f79761576c580607efbdca95a0ae6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:50:56.843 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90508]: 2024-03-28 18:50:56.572536218 +0000 UTC m=+0.020512478 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:50:56.843 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90508]: 2024-03-28 18:50:56.714657377 +0000 UTC m=+0.162633635 container create 991aa009d6e38cc96561f164e0fddcf508be0fa8de95a5d6685add601294c0e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:50:57.277 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90508]: 2024-03-28 18:50:56.984139443 +0000 UTC m=+0.432115702 container init 991aa009d6e38cc96561f164e0fddcf508be0fa8de95a5d6685add601294c0e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:50:57.277 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:56 smithi145 podman[90508]: 2024-03-28 18:50:56.987346772 +0000 UTC m=+0.435323032 container start 991aa009d6e38cc96561f164e0fddcf508be0fa8de95a5d6685add601294c0e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T18:50:57.277 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:57 smithi145 podman[90508]: 2024-03-28 18:50:57.084420899 +0000 UTC m=+0.532397204 container attach 991aa009d6e38cc96561f164e0fddcf508be0fa8de95a5d6685add601294c0e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T18:50:57.871 INFO:journalctl@ceph.osd.0.smithi145.stdout:Mar 28 18:50:57 smithi145 podman[90508]: 2024-03-28 18:50:57.546638301 +0000 UTC m=+0.994614554 container died 991aa009d6e38cc96561f164e0fddcf508be0fa8de95a5d6685add601294c0e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T18:50:58.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T18:50:58.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:50:58.236 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T18:50:58.236 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T18:50:58.236 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.1 2024-03-28T18:50:58.567 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:50:58 smithi164 systemd[1]: Stopping Ceph osd.1 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:50:58.962 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:50:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1[40830]: 2024-03-28T18:50:58.566+0000 7f3cf7d65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:50:58.962 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:50:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1[40830]: 2024-03-28T18:50:58.566+0000 7f3cf7d65700 -1 osd.1 99 *** Got signal Terminated *** 2024-03-28T18:50:58.962 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:50:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1[40830]: 2024-03-28T18:50:58.566+0000 7f3cf7d65700 -1 osd.1 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:04.462 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:04 smithi164 podman[71533]: 2024-03-28 18:51:04.068891999 +0000 UTC m=+5.632390990 container died 95f2b6ac42b8f9c1ac03e243842f03681fee71b0465307650c0fb100bd084952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:04.962 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:04 smithi164 podman[71533]: 2024-03-28 18:51:04.793614781 +0000 UTC m=+6.357113765 container cleanup 95f2b6ac42b8f9c1ac03e243842f03681fee71b0465307650c0fb100bd084952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:04.963 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:04 smithi164 bash[71533]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1 2024-03-28T18:51:05.432 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71549]: 2024-03-28 18:51:05.028236568 +0000 UTC m=+0.958607607 container remove 95f2b6ac42b8f9c1ac03e243842f03681fee71b0465307650c0fb100bd084952 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:51:05.712 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71643]: 2024-03-28 18:51:05.336453379 +0000 UTC m=+0.017837861 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:05.712 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71643]: 2024-03-28 18:51:05.462555979 +0000 UTC m=+0.143940443 container create 7155eb074ac7c3855a8e0665bc5375fe41404a1fc93a73fa50cf7ca38d25f7c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T18:51:06.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71643]: 2024-03-28 18:51:05.757710365 +0000 UTC m=+0.439094831 container init 7155eb074ac7c3855a8e0665bc5375fe41404a1fc93a73fa50cf7ca38d25f7c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:06.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71643]: 2024-03-28 18:51:05.760953191 +0000 UTC m=+0.442337658 container start 7155eb074ac7c3855a8e0665bc5375fe41404a1fc93a73fa50cf7ca38d25f7c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T18:51:06.212 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:05 smithi164 podman[71643]: 2024-03-28 18:51:05.89988536 +0000 UTC m=+0.581269907 container attach 7155eb074ac7c3855a8e0665bc5375fe41404a1fc93a73fa50cf7ca38d25f7c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T18:51:06.771 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 28 18:51:06 smithi164 podman[71643]: 2024-03-28 18:51:06.508230988 +0000 UTC m=+1.189615453 container died 7155eb074ac7c3855a8e0665bc5375fe41404a1fc93a73fa50cf7ca38d25f7c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:07.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T18:51:07.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:07.174 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T18:51:07.175 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T18:51:07.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.2 2024-03-28T18:51:07.481 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:07 smithi192 systemd[1]: Stopping Ceph osd.2 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:07.741 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2[41773]: 2024-03-28T18:51:07.480+0000 7f3104cc6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:07.741 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2[41773]: 2024-03-28T18:51:07.480+0000 7f3104cc6700 -1 osd.2 99 *** Got signal Terminated *** 2024-03-28T18:51:07.741 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2[41773]: 2024-03-28T18:51:07.480+0000 7f3104cc6700 -1 osd.2 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:13.459 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:13 smithi192 podman[71892]: 2024-03-28 18:51:13.166445885 +0000 UTC m=+5.803043661 container died 7812b7a2ca01f41140e0f9491e4e0d2e1c9ecb52ab8cc6a821c6959aeff1c800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:13.859 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:13 smithi192 podman[71892]: 2024-03-28 18:51:13.541975647 +0000 UTC m=+6.178573416 container cleanup 7812b7a2ca01f41140e0f9491e4e0d2e1c9ecb52ab8cc6a821c6959aeff1c800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T18:51:13.860 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:13 smithi192 bash[71892]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2 2024-03-28T18:51:14.157 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:13 smithi192 podman[71921]: 2024-03-28 18:51:13.860318665 +0000 UTC m=+0.686001138 container remove 7812b7a2ca01f41140e0f9491e4e0d2e1c9ecb52ab8cc6a821c6959aeff1c800 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:51:14.438 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:14 smithi192 podman[72010]: 2024-03-28 18:51:14.069472553 +0000 UTC m=+0.018011783 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:14.439 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:14 smithi192 podman[72010]: 2024-03-28 18:51:14.186168323 +0000 UTC m=+0.134707542 container create 9ef35f4c7d480721be26e5c3b89b48178ffd2b03b69e51effc43c829b332fb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:51:14.907 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:14 smithi192 podman[72010]: 2024-03-28 18:51:14.438929357 +0000 UTC m=+0.387468578 container init 9ef35f4c7d480721be26e5c3b89b48178ffd2b03b69e51effc43c829b332fb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:14.907 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:14 smithi192 podman[72010]: 2024-03-28 18:51:14.442800829 +0000 UTC m=+0.391340058 container start 9ef35f4c7d480721be26e5c3b89b48178ffd2b03b69e51effc43c829b332fb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T18:51:14.907 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:14 smithi192 podman[72010]: 2024-03-28 18:51:14.572243905 +0000 UTC m=+0.520783142 container attach 9ef35f4c7d480721be26e5c3b89b48178ffd2b03b69e51effc43c829b332fb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T18:51:15.385 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 28 18:51:15 smithi192 podman[72010]: 2024-03-28 18:51:15.0350589 +0000 UTC m=+0.983598130 container died 9ef35f4c7d480721be26e5c3b89b48178ffd2b03b69e51effc43c829b332fb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T18:51:15.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T18:51:15.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:15.757 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T18:51:15.757 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T18:51:15.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.3 2024-03-28T18:51:16.074 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:15 smithi145 systemd[1]: Stopping Ceph osd.3 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:16.526 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:16 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:16.073+0000 7efda7954700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:16.527 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:16 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:16.073+0000 7efda7954700 -1 osd.3 99 *** Got signal Terminated *** 2024-03-28T18:51:16.527 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:16 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:16.073+0000 7efda7954700 -1 osd.3 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:16.372+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:17.276 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:16 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:16.978+0000 7efda3f48700 -1 osd.3 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.727236+0000 front 2024-03-28T18:50:53.727263+0000 (oldest deadline 2024-03-28T18:51:16.627464+0000) 2024-03-28T18:51:17.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:16 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:16.839+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:17.380+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:18.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:17.846+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:18.276 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:17 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:17.929+0000 7efda3f48700 -1 osd.3 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.727236+0000 front 2024-03-28T18:50:53.727263+0000 (oldest deadline 2024-03-28T18:51:16.627464+0000) 2024-03-28T18:51:18.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:17 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:17.840+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:18.364+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:19.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:18.858+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:19.276 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:18 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:18.944+0000 7efda3f48700 -1 osd.3 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.727236+0000 front 2024-03-28T18:50:53.727263+0000 (oldest deadline 2024-03-28T18:51:16.627464+0000) 2024-03-28T18:51:19.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:18 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:18.827+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:19.320+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:20.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:19.893+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:20.276 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:19 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:19.903+0000 7efda3f48700 -1 osd.3 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.727236+0000 front 2024-03-28T18:50:53.727263+0000 (oldest deadline 2024-03-28T18:51:16.627464+0000) 2024-03-28T18:51:20.314 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:19 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:19.808+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:20.324+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:21.276 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:20 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3[50798]: 2024-03-28T18:51:20.933+0000 7efda3f48700 -1 osd.3 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.727236+0000 front 2024-03-28T18:50:53.727263+0000 (oldest deadline 2024-03-28T18:51:16.627464+0000) 2024-03-28T18:51:21.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:20 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:20.843+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:21.357 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:20.933+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:21.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:21.356+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:21.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:21 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:21.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:22.073 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:21 smithi145 podman[90614]: 2024-03-28 18:51:21.784515478 +0000 UTC m=+5.834534715 container died c6cdcedd03a79d42528ff1c5bccde8fa2653559cbc3a15cf43b9e6f3ef877433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T18:51:22.073 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:21 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:21.892+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:22.383 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:21.909+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:22.442 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:22 smithi145 podman[90614]: 2024-03-28 18:51:22.157632652 +0000 UTC m=+6.207651876 container cleanup c6cdcedd03a79d42528ff1c5bccde8fa2653559cbc3a15cf43b9e6f3ef877433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:22.442 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:22 smithi145 bash[90614]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3 2024-03-28T18:51:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:22.383+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:22.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:22 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:22.369+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:22.774 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:22 smithi145 podman[90645]: 2024-03-28 18:51:22.442507203 +0000 UTC m=+0.648280984 container remove c6cdcedd03a79d42528ff1c5bccde8fa2653559cbc3a15cf43b9e6f3ef877433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:51:23.026 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:22 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:22.900+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:23.027 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:22 smithi145 podman[90740]: 2024-03-28 18:51:22.678524114 +0000 UTC m=+0.017999348 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:23.027 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:22 smithi145 podman[90740]: 2024-03-28 18:51:22.793511142 +0000 UTC m=+0.132986366 container create 729b03c94825322b182fe6dd75ea1e480d9a194023880a82fe4387eaef0803b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:51:23.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:22.893+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:23.382 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:23 smithi145 podman[90740]: 2024-03-28 18:51:23.079466862 +0000 UTC m=+0.418942085 container init 729b03c94825322b182fe6dd75ea1e480d9a194023880a82fe4387eaef0803b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T18:51:23.383 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:23 smithi145 podman[90740]: 2024-03-28 18:51:23.08271408 +0000 UTC m=+0.422189305 container start 729b03c94825322b182fe6dd75ea1e480d9a194023880a82fe4387eaef0803b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T18:51:23.383 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:23 smithi145 podman[90740]: 2024-03-28 18:51:23.179470838 +0000 UTC m=+0.518946082 container attach 729b03c94825322b182fe6dd75ea1e480d9a194023880a82fe4387eaef0803b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T18:51:23.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:23.339+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:23.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:23 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:23.332+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:23.741 INFO:journalctl@ceph.osd.3.smithi145.stdout:Mar 28 18:51:23 smithi145 podman[90740]: 2024-03-28 18:51:23.383451959 +0000 UTC m=+0.722927187 container died 729b03c94825322b182fe6dd75ea1e480d9a194023880a82fe4387eaef0803b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:24.000 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:23 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:23.941+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:24.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T18:51:24.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:24.063 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T18:51:24.064 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T18:51:24.064 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.4 2024-03-28T18:51:24.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:23.874+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:24.369 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:24 smithi164 systemd[1]: Stopping Ceph osd.4 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:24.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:24.348+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:24.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:24.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:24.712 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:24.389+0000 7f40e4da1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:24.713 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:24.389+0000 7f40e4da1700 -1 osd.4 99 *** Got signal Terminated *** 2024-03-28T18:51:24.713 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:24.389+0000 7f40e4da1700 -1 osd.4 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:25.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:24.900+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:25.212 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:24 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:24.960+0000 7f40dd38d700 -1 osd.4 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.422316+0000 front 2024-03-28T18:51:02.422337+0000 (oldest deadline 2024-03-28T18:51:24.722478+0000) 2024-03-28T18:51:25.276 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:24 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:24.963+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:25.386+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:25.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:25 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:25.521+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:25.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:25 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:25.323+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:25.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:25 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:25.323+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:26.276 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:25 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:25.950+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:26.300 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:25 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:25.990+0000 7f40dd38d700 -1 osd.4 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.422316+0000 front 2024-03-28T18:51:02.422337+0000 (oldest deadline 2024-03-28T18:51:24.722478+0000) 2024-03-28T18:51:26.385 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:25.914+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:26.386+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:26.386+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:26.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:26 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:26.474+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:26.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:26 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:26.300+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:26.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:26 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:26.300+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:27.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:26.903+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:27.276 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:26 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:26.961+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:27.341 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:26 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:26.992+0000 7f40dd38d700 -1 osd.4 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.422316+0000 front 2024-03-28T18:51:02.422337+0000 (oldest deadline 2024-03-28T18:51:24.722478+0000) 2024-03-28T18:51:27.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:27 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:27.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:27.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:27 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:27.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:27.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:27 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:27.506+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:27.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:27 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:27.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:27.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:27 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:27.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:28.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:27 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:27.901+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:28.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:28 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:28.056+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:28.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:27 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:27.975+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:28.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:27 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:27.929+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:28.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:27 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:27.929+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:28.342 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:27 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:27.990+0000 7f40dd38d700 -1 osd.4 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.422316+0000 front 2024-03-28T18:51:02.422337+0000 (oldest deadline 2024-03-28T18:51:24.722478+0000) 2024-03-28T18:51:28.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:28 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:28.324+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:28.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:28 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:28.324+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:28.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:28 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:28.540+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:28.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:28 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:28.342+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:28.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:28 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:28.342+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:29.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:29.064+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:29.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:28 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:28.865+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:29.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:28 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:28.865+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:29.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:28.981+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:29.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:28.976+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:29.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:28 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:28.976+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:29.294 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:28 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4[47533]: 2024-03-28T18:51:28.969+0000 7f40dd38d700 -1 osd.4 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.422316+0000 front 2024-03-28T18:51:02.422337+0000 (oldest deadline 2024-03-28T18:51:24.722478+0000) 2024-03-28T18:51:29.588 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:29 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:29.294+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:29.588 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:29 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:29.294+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:29.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:29.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:29.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:29.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:29.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:29 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:29.588+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:30.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:30 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:30.055+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:30.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:29.886+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:30.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:29 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:29.886+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:30.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:29 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:29.940+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:30.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:30 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:30.017+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:30.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:30 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:30.017+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:30.293 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:30 smithi164 podman[71764]: 2024-03-28 18:51:30.005574954 +0000 UTC m=+5.743829271 container died d5bc7f950bbf49be87d2be1fa29ae8d571447843a9c37224f1565bdd92cd8222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T18:51:30.591 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:30 smithi164 podman[71764]: 2024-03-28 18:51:30.423333372 +0000 UTC m=+6.161587717 container cleanup d5bc7f950bbf49be87d2be1fa29ae8d571447843a9c37224f1565bdd92cd8222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T18:51:30.591 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:30 smithi164 bash[71764]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4 2024-03-28T18:51:30.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:30 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:30.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:30.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:30 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:30.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:30.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:30 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:30.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:30.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:30 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:30.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:30.962 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:30 smithi164 podman[71783]: 2024-03-28 18:51:30.666466791 +0000 UTC m=+0.665951416 container remove d5bc7f950bbf49be87d2be1fa29ae8d571447843a9c37224f1565bdd92cd8222 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:51:30.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:30 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:30.590+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:31.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:31 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:31.073+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:31.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:30 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:30.879+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:31.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:30 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:30.879+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:31.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:30 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:30.914+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:31.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:31 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:31.018+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:31.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:31 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:31.018+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:31.333 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:30.967952254 +0000 UTC m=+0.018180999 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:31.333 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:31.075658372 +0000 UTC m=+0.125887104 container create 7a24f904496ce63880a4272e7fd5268e00af8ef35b3940346f2671ca2d912ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-28T18:51:31.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:31 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:31.348+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:31.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:31 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:31.348+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:31.678 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:31 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:31.543+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:31.678 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:31 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:31.332+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:31.678 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:31 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:31.332+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:31.679 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:31.386448568 +0000 UTC m=+0.436677299 container init 7a24f904496ce63880a4272e7fd5268e00af8ef35b3940346f2671ca2d912ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-28T18:51:31.679 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:31.389673759 +0000 UTC m=+0.439902492 container start 7a24f904496ce63880a4272e7fd5268e00af8ef35b3940346f2671ca2d912ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:31.679 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:31.494961551 +0000 UTC m=+0.545190293 container attach 7a24f904496ce63880a4272e7fd5268e00af8ef35b3940346f2671ca2d912ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T18:51:31.962 INFO:journalctl@ceph.osd.4.smithi164.stdout:Mar 28 18:51:31 smithi164 podman[71875]: 2024-03-28 18:51:31.679146087 +0000 UTC m=+0.729374835 container died 7a24f904496ce63880a4272e7fd5268e00af8ef35b3940346f2671ca2d912ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T18:51:32.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:32.050+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:32.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:31 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:31.884+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:32.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:31 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:31.884+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:32.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:31 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:31.864+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:32.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:32 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:32.002+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:32.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:32 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:32.002+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:32.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T18:51:32.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:32.362 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T18:51:32.362 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T18:51:32.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.5 2024-03-28T18:51:32.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:32 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:32.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:32.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:32 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:32.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:32.657 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 systemd[1]: Stopping Ceph osd.5 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:32.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:32.349+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:32.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:32.349+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:32.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:32 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:32.591+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:33.082 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:32.696+0000 7f68e0fd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:33.082 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:32.696+0000 7f68e0fd8700 -1 osd.5 99 *** Got signal Terminated *** 2024-03-28T18:51:33.082 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:32.696+0000 7f68e0fd8700 -1 osd.5 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:33.082 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:32.908+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:33.082 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:32 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:32.908+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:33.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:32 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:32.907+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:33.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:33.045+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:33.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:33.045+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:33.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:33 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:33.081+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:33.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:33 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:33.315+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:33.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:33 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:33.315+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:33.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:33 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:33.586+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:33.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:33 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:33.370+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:33.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:33 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:33.370+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:34.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:33 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:33.882+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:34.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:33 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:33.882+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:34.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:34.114+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:34.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:34.114+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:34.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:33 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:33.900+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:34.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:34.017+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:34.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:34.017+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:34.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:34.017+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:34.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:34.346+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:34.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:34.346+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:34.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:34 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:34.552+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:34.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:34 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:34.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:34.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:34 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:34.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:35.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:34.864+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:35.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:34.864+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:35.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:35 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:35.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:35.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:35 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:35.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:35.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:35 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:35.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:35.351 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:34.930+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:35.351 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:34 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:34.930+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:35.351 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:35.148+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:35.351 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:35.149+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:35.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:35.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:35.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:35.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:35.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:35.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:35.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:35 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:35.569+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:35.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:35 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:35.382+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:35.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:35 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:35.382+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:36.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:35 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:35.883+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:36.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:35 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:35.883+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:36.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:36.067+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:36.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:36.067+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:36.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:36.067+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:36.333 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:35.916+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:36.333 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:35 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:35.916+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:36.333 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:36.166+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:36.333 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:36.166+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:36.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:36.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:36.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:36.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:36.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:36.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:36.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:36 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:36.345+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:36.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:36 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:36.345+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:36.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:36 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:36.581+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:36.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:36 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:36.581+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:37.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:36.855+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:37.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:36 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:36.855+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:37.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:37.072+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:37.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:37.072+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:37.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:37.072+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:37.349 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:36.964+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:54.508283+0000 front 2024-03-28T18:50:54.508249+0000 (oldest deadline 2024-03-28T18:51:17.408451+0000) 2024-03-28T18:51:37.349 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:36 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5[48279]: 2024-03-28T18:51:36.964+0000 7f68d95c4700 -1 osd.5 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.010095+0000 front 2024-03-28T18:51:02.010134+0000 (oldest deadline 2024-03-28T18:51:27.910254+0000) 2024-03-28T18:51:37.349 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:37.206+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:37.349 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:37.206+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:37.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:37.313+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:37.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:37.313+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:37.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:37.313+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:37.630 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:37.349+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:37.630 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:37.349+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:37.630 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:37 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:37.349+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:37.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:37.613+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:37.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:37 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:37.613+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:38.276 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:37.871+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:38.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:37 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:37.871+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:38.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:38.093+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:38.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:38.093+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:38.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:38.093+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:38.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:38 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:38.368+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:38.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:38 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:38.369+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:38.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:38 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:38.369+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:38.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:38 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:38.158+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:38.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:38 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:38.158+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:38.601 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:38 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:38.299+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:38.601 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:38 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:38.299+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:38.602 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:38 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:38.299+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:38.821 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:38 smithi192 podman[72115]: 2024-03-28 18:51:38.545603383 +0000 UTC m=+5.984509018 container died c0fdfbd31606e87aa5b23166144302e9cbeac2c1fa460ff82ca913d844258137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:51:38.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:38 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:38.601+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:38.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:38 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:38.601+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:39.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:38 smithi192 podman[72115]: 2024-03-28 18:51:38.928663877 +0000 UTC m=+6.367569510 container cleanup c0fdfbd31606e87aa5b23166144302e9cbeac2c1fa460ff82ca913d844258137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:39.157 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:38 smithi192 bash[72115]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5 2024-03-28T18:51:39.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:38.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:39.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:38.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:39.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:39 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:39.132+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:39.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:39 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:39.132+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:39.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:39 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:39.132+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72142]: 2024-03-28 18:51:39.230325569 +0000 UTC m=+0.679449065 container remove c0fdfbd31606e87aa5b23166144302e9cbeac2c1fa460ff82ca913d844258137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:39 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:39.171+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:39 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:39.171+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:39 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:39.392+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:39 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:39.392+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:39.527 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:39 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:39.392+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:39.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:39 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:39.576+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:39.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:39 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:39.576+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:39.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:39 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:39.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:39.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:39 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:39.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:39.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:39 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:39.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:39.778 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72241]: 2024-03-28 18:51:39.430720933 +0000 UTC m=+0.018360462 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:39.778 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72241]: 2024-03-28 18:51:39.573029043 +0000 UTC m=+0.160668552 container create ff0a063aa6f0424c06e0616a41b2204899e301c8b35deeb8e6c0f47e1780f783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:51:39.778 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72241]: 2024-03-28 18:51:39.775690527 +0000 UTC m=+0.363330050 container init ff0a063aa6f0424c06e0616a41b2204899e301c8b35deeb8e6c0f47e1780f783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-28T18:51:39.778 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72241]: 2024-03-28 18:51:39.778970335 +0000 UTC m=+0.366609842 container start ff0a063aa6f0424c06e0616a41b2204899e301c8b35deeb8e6c0f47e1780f783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:40.070 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:39 smithi192 podman[72241]: 2024-03-28 18:51:39.875403446 +0000 UTC m=+0.463042982 container attach ff0a063aa6f0424c06e0616a41b2204899e301c8b35deeb8e6c0f47e1780f783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T18:51:40.159 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:39 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:39.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:40.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:39 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:39.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:40.356 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 28 18:51:40 smithi192 podman[72241]: 2024-03-28 18:51:40.07163598 +0000 UTC m=+0.659275496 container died ff0a063aa6f0424c06e0616a41b2204899e301c8b35deeb8e6c0f47e1780f783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:51:40.357 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:40 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:40.169+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:40.357 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:40 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:40.169+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:40.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:40 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:40.159+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:40.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:40 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:40.159+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:40.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:40 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:40.159+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:40.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:40 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:40.354+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:40.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:40 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:40.355+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:40.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:40 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:40.355+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:40.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:40 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:40.542+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:40.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:40 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:40.542+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:40.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:40 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:40.351+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:40.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:40 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:40.351+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:40.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:40 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:40.351+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:40.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T18:51:40.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:40.851 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T18:51:40.851 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T18:51:40.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.6 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:40 smithi145 systemd[1]: Stopping Ceph osd.6 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:40 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:40.917+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:40 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:40.917+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:41.160+0000 7f0cd9113700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:41.160+0000 7f0cd9113700 -1 osd.6 99 *** Got signal Terminated *** 2024-03-28T18:51:41.160 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:41.160+0000 7f0cd9113700 -1 osd.6 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:41.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:41.153+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:41.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:41.153+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:41.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:41.315+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:41.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:41.315+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:41.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:41 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:41.315+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:41.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:41.201+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:41.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:41.201+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:41.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:41.201+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:41.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:41 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:41.541+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:41.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:41 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:41.541+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:41.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:41 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:41.383+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:41.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:41 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:41.383+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:41.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:41 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:41.383+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:42.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:41.951+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:42.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:41 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:41.951+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:42.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:42.182+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:42.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:42.182+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:42.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:42.182+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:42.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:42 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:42.182+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:42.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:42 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:42.182+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:42.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:42 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:42.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:42.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:42 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:42.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:42.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:42 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:42.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:42.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:42 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:42.530+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:42.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:42 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:42.530+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:42.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:42 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:42.396+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:42.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:42 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:42.396+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:42.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:42 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:42.396+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:43.164 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:42.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:43.164 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:42.908+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:43.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:43 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:43.152+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:43.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:43 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:43.152+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:43.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:43 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:43.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:43.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:43 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:43.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:43.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:43 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:43.365+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:43.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:43.164+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:43.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:43.164+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:43.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:43.164+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:43.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:43.558+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:43.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:43.558+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:43.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:43.374+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:43.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:43.374+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:43.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:43.374+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:43.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:43 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:43.374+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:43.940+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:43.940+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:43 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:43.940+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.185695+0000 front 2024-03-28T18:51:18.185634+0000 (oldest deadline 2024-03-28T18:51:43.485827+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:44.126+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:44.126+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:44.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:44.126+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:44.376+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:44.376+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:44.376+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:44.198+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:44.198+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:44.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:44 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:44.198+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:44.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:44.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:44.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:44.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:44.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:44.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:44.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:44.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:44.713 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:44.510+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:44.713 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:44.510+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:44.713 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:44 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:44.510+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:45.176 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:44.922+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:45.177 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:44.922+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:45.177 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:44 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:44.922+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.185695+0000 front 2024-03-28T18:51:18.185634+0000 (oldest deadline 2024-03-28T18:51:43.485827+0000) 2024-03-28T18:51:45.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:45.176+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:45.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:45.176+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:45.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:45.176+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:45.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:45.176+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:45.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:45.201+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:45.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:45.201+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:45.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:45.201+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:45.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:45.372+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:45.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:45.372+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:45.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:45 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:45.372+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:45.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:45.536+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:45.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:45.536+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:45.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:45.536+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:45.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:45.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:45.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:45.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:45.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:45.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:45.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:45 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:45.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:46.206 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:45.903+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:02.383966+0000 front 2024-03-28T18:51:02.383782+0000 (oldest deadline 2024-03-28T18:51:27.683982+0000) 2024-03-28T18:51:46.206 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:45.903+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:08.785292+0000 front 2024-03-28T18:51:08.785052+0000 (oldest deadline 2024-03-28T18:51:34.684899+0000) 2024-03-28T18:51:46.206 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:45 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6[57316]: 2024-03-28T18:51:45.903+0000 7f0cd5707700 -1 osd.6 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.185695+0000 front 2024-03-28T18:51:18.185634+0000 (oldest deadline 2024-03-28T18:51:43.485827+0000) 2024-03-28T18:51:46.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:46.205+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:46.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:46.205+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:46.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:46.205+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:46.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:46 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:46.205+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:46.209+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:46.209+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:46.209+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:46.378+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:46.378+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:46.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:46 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:46.378+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:46.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:46.546+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:46.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:46.546+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:46.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:46.546+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:46.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:46.316+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:46.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:46.316+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:46.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:46.316+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:46.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:46 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:46.316+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:47.214 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:46 smithi145 podman[90845]: 2024-03-28 18:51:46.898542294 +0000 UTC m=+5.851218558 container died 90e78dd2ccb534df8729df495ad373af69cc2954540c5e11699b60a758bf648e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:47.213+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:47.213+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:47.213+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:47 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:47.213+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:47 smithi145 podman[90845]: 2024-03-28 18:51:47.235103703 +0000 UTC m=+6.187779975 container cleanup 90e78dd2ccb534df8729df495ad373af69cc2954540c5e11699b60a758bf648e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:47 smithi145 bash[90845]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6 2024-03-28T18:51:47.527 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:47 smithi145 podman[90863]: 2024-03-28 18:51:47.411758904 +0000 UTC m=+0.518067549 container remove 90e78dd2ccb534df8729df495ad373af69cc2954540c5e11699b60a758bf648e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:51:47.613 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:47.538+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:47.613 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:47.538+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:47.613 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:47.538+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:47.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:47.355+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:47.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:47.355+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:47.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:47.355+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:47.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:47.355+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:47.191+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:47.191+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:47.191+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:47.394+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:47.394+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:47.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:47 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:47.394+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:48.187 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:47 smithi145 podman[90962]: 2024-03-28 18:51:47.790371228 +0000 UTC m=+0.018010719 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:48.187 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:47 smithi145 podman[90962]: 2024-03-28 18:51:47.92117193 +0000 UTC m=+0.148811411 container create 9b01902e5b05fadf75ab6dba4829c15db86b7037c5c6acb0e04902027436976e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T18:51:48.403 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:48.204+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:48.403 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:48.204+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:48.403 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:48.204+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:48.519 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:48.186+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:48.519 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:48.186+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:48.519 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:48.186+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:48.519 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:48 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:48.186+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:48.520 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:48 smithi145 podman[90962]: 2024-03-28 18:51:48.224184738 +0000 UTC m=+0.451824220 container init 9b01902e5b05fadf75ab6dba4829c15db86b7037c5c6acb0e04902027436976e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T18:51:48.520 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:48 smithi145 podman[90962]: 2024-03-28 18:51:48.227257162 +0000 UTC m=+0.454896644 container start 9b01902e5b05fadf75ab6dba4829c15db86b7037c5c6acb0e04902027436976e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T18:51:48.520 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:48 smithi145 podman[90962]: 2024-03-28 18:51:48.307259533 +0000 UTC m=+0.534899026 container attach 9b01902e5b05fadf75ab6dba4829c15db86b7037c5c6acb0e04902027436976e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T18:51:48.520 INFO:journalctl@ceph.osd.6.smithi145.stdout:Mar 28 18:51:48 smithi145 podman[90962]: 2024-03-28 18:51:48.519897711 +0000 UTC m=+0.747537201 container died 9b01902e5b05fadf75ab6dba4829c15db86b7037c5c6acb0e04902027436976e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:51:48.627 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:48.541+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:48.627 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:48.541+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:48.627 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:48.541+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:48.627 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:48.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:48.628 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:48.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:48.628 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:48.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:48.628 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:48 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:48.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:48.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:48.402+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:48.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:48.402+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:48.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:48 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:48.402+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:49.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T18:51:49.252 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:49.252 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T18:51:49.252 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T18:51:49.252 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.7 2024-03-28T18:51:49.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:49.232+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:49.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:49.232+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:49.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:49.232+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:49.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:49 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:49.232+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:49.544 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 systemd[1]: Stopping Ceph osd.7 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:49.545 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:49.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:49.545 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:49.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:49.545 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:49.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:49.545 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:49.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:49.373+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:49.373+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:49.373+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:49.185+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:49.185+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:49.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:49 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:49.185+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.544+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.544+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.544+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.584+0000 7fc6935b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.584+0000 7fc6935b2700 -1 osd.7 99 *** Got signal Terminated *** 2024-03-28T18:51:49.848 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:49 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:49.584+0000 7fc6935b2700 -1 osd.7 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:50.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:50.370+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:50.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:50.370+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:50.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:50.370+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:50.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:50.140+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:50.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:50.140+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:50.408 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:50 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:50.140+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:50.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:50.210+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:50.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:50.210+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:50.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:50.210+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:50.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:50 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:50.210+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:50.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:50.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:50.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:50.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:50.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:50.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:50.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:50.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:50.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:50.554+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:50.713 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:50.554+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:50.713 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:50.554+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:51.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:51.100+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:51.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:51.100+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:51.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:51.100+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:51.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:51.181+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:51.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:51.181+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:51.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:51.181+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:51.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:51 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:51.181+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:51.601 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:51.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:51.601 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:51.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:51.602 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:51.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:51.602 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:51.340+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:51.907 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:51.415+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:51.907 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:51.415+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:51.907 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:51 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:51.415+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:51.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:51.601+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:51.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:51.601+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:51.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:51 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:51.601+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:52.381 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:52.087+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:52.381 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:52.087+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:52.381 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:52.087+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:52.381 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:52.087+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:52.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:52 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:52.145+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:52.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:52 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:52.145+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:52.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:52 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:52.145+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:52.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:52 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:52.145+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:52.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:52.381+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:52.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:52.381+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:52.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:52 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:52.381+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:52.562+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:52.562+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:52.562+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:52.357+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:52.357+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:52.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:52.357+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:52.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:52 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:52.357+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:53.334 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:53.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:53.335 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:53.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:53.335 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:53.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:53.335 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:53.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:53.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:53.148+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:53.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:53.148+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:53.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:53.148+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:53.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:53 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:53.148+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:53.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:53.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:53.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:53.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:53.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:53.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:53.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:53 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:53.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:53.571+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:50:59.115095+0000 front 2024-03-28T18:50:59.115120+0000 (oldest deadline 2024-03-28T18:51:25.015325+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:53.571+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.117629+0000 front 2024-03-28T18:51:10.117478+0000 (oldest deadline 2024-03-28T18:51:36.017621+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:53.571+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:16.018181+0000 front 2024-03-28T18:51:16.018215+0000 (oldest deadline 2024-03-28T18:51:43.618730+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7[54148]: 2024-03-28T18:51:53.571+0000 7fc68bb9e700 -1 osd.7 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:24.119682+0000 front 2024-03-28T18:51:24.119642+0000 (oldest deadline 2024-03-28T18:51:52.920273+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:53.388+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:53.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:53.388+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:53.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:53.388+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:53.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:53.388+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:53.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:53 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:53.388+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:54.285+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:54.285+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:54.285+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:54.285+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:54.091+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:54.091+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:54.091+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:54.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:54 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:54.091+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:54.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:54 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:54.136+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:54.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:54 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:54.136+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:54.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:54 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:54.136+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:54.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:54 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:54.136+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:54.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:54.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:54.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:54.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:54.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:54.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:54.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:54.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:54.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:54.368+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:55.057+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:55.057+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:55.057+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:55.057+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:55.266+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:55.266+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:55.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:55.266+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:55.408 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:55 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:55.266+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:55.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:55 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:55.107+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:55.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:55 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:55.107+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:55.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:55 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:55.107+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:55.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:55 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:55.107+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:55.634 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:55 smithi164 podman[71991]: 2024-03-28 18:51:55.341976788 +0000 UTC m=+5.880550187 container died 69946ceb4e2a6cb885c0a9607e1a4980b47f53448a127fa7118474adb0e4500a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T18:51:55.635 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:55 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:55.349+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:55.635 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:55 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:55.349+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:55.635 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:55 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:55.349+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:55.635 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:55 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:55.349+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:55.635 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:55 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:55.349+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:55.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:55 smithi164 podman[71991]: 2024-03-28 18:51:55.726136662 +0000 UTC m=+6.264710036 container cleanup 69946ceb4e2a6cb885c0a9607e1a4980b47f53448a127fa7118474adb0e4500a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T18:51:55.962 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:55 smithi164 bash[71991]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7 2024-03-28T18:51:56.315 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72009]: 2024-03-28 18:51:56.002403741 +0000 UTC m=+0.662566915 container remove 69946ceb4e2a6cb885c0a9607e1a4980b47f53448a127fa7118474adb0e4500a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:56.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:56.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:56.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:56.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:56.273+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:56.273+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:56.273+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:56.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:56 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:56.273+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:56.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:56 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:56.111+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:56.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:56 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:56.111+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:56.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:56 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:56.111+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:56.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:56 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:56.111+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:56.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:56 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:56.111+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:56 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:56.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:56 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:56.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:56 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:56.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:56 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:56.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:56 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:56.366+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.21927944 +0000 UTC m=+0.017951663 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.345382029 +0000 UTC m=+0.144054279 container create 48ec8439d5e253b345b097baa01c3120f2199a680a9df4e07b948dc5f998e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.564485251 +0000 UTC m=+0.363157466 container init 48ec8439d5e253b345b097baa01c3120f2199a680a9df4e07b948dc5f998e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:51:56.567 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.567675625 +0000 UTC m=+0.366347826 container start 48ec8439d5e253b345b097baa01c3120f2199a680a9df4e07b948dc5f998e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:51:56.865 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.67300271 +0000 UTC m=+0.471674925 container attach 48ec8439d5e253b345b097baa01c3120f2199a680a9df4e07b948dc5f998e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:51:57.212 INFO:journalctl@ceph.osd.7.smithi164.stdout:Mar 28 18:51:56 smithi164 podman[72101]: 2024-03-28 18:51:56.865587591 +0000 UTC m=+0.664259809 container died 48ec8439d5e253b345b097baa01c3120f2199a680a9df4e07b948dc5f998e2b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.059+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:57.299+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:57.299+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:57.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:57.299+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:57.408 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:57.299+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:57.501 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:57 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:57.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:57.501 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:57 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:57.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:57.501 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:57 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:57.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:57.501 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:57 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:57.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:57.501 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:57 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:57.385+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:57.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T18:51:57.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:57 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:57.076+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:57.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:57 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:57.076+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:57.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:57 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:57.076+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:57.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:57 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:57.076+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:57.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:57 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:57.076+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:51:57.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:51:57.559 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T18:51:57.559 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T18:51:57.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.8 2024-03-28T18:51:57.874 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 systemd[1]: Stopping Ceph osd.8 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:51:57.874 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.873+0000 7ff617b37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:51:57.874 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.873+0000 7ff617b37700 -1 osd.8 99 *** Got signal Terminated *** 2024-03-28T18:51:57.874 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:57 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:57.873+0000 7ff617b37700 -1 osd.8 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:51:58.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:58.046+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:58.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:58.046+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:58.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:58.046+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:58.157 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:58.046+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:58.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:58 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:58.057+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:58.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:58 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:58.057+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:58.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:58 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:58.057+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:58.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:58 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:58.057+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:58.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:58 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:58.057+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:51:58.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:58.261+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:58.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:58.261+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:58.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:58.261+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:58.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:58 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:58.261+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:58.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:58.352+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:58.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:58.352+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:58.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:58.352+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:58.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:58.352+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:58.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:58.352+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:59.247+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:59.247+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:59.247+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:51:59.247+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:59.080+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:59.080+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:59.080+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:51:59.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:51:59 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:51:59.080+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:51:59.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:59 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:59.027+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:51:59.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:59 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:59.027+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:51:59.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:59 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:59.027+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:51:59.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:59 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:59.027+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:51:59.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:51:59 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:51:59.027+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:51:59.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:59 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:59.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:51:59.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:59 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:59.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:51:59.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:59 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:59.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:51:59.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:59 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:59.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:51:59.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:51:59 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:51:59.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:00.072+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:00.072+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:00.072+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:00.072+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:00.295+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:00.295+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:00.407 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:00.295+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:00.408 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:00 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:00.295+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:00.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:00 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:00.069+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:00.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:00 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:00.069+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:00.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:00 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:00.069+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:00.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:00 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:00.069+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:00.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:00 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:00.069+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:00.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:00 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:00.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:00.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:00 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:00.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:00.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:00 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:00.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:00.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:00 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:00.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:00.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:00 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:00.346+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:01.334 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:01.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:52:01.334 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:01.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:52:01.334 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:01.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:52:01.335 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:01.066+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:52:01.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:01 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:01.115+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:01.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:01 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:01.115+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:01.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:01 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:01.115+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:01.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:01 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:01.115+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:01.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:01 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:01.115+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:01.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:01.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:01.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:01.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:01.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:01.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:01.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:01 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:01.334+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:01.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:01 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:01.329+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:02.379 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:02.033+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:03.312807+0000 front 2024-03-28T18:51:03.313009+0000 (oldest deadline 2024-03-28T18:51:28.012900+0000) 2024-03-28T18:52:02.379 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:02.033+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.113767+0000 front 2024-03-28T18:51:09.113746+0000 (oldest deadline 2024-03-28T18:51:33.813891+0000) 2024-03-28T18:52:02.379 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:02.033+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:19.114819+0000 front 2024-03-28T18:51:19.114899+0000 (oldest deadline 2024-03-28T18:51:43.814740+0000) 2024-03-28T18:52:02.380 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:02.033+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.115778+0000 front 2024-03-28T18:51:29.115695+0000 (oldest deadline 2024-03-28T18:51:52.015654+0000) 2024-03-28T18:52:02.380 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8[54830]: 2024-03-28T18:52:02.033+0000 7ff610123700 -1 osd.8 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.316645+0000 front 2024-03-28T18:51:37.316595+0000 (oldest deadline 2024-03-28T18:52:01.416689+0000) 2024-03-28T18:52:02.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:02 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:02.119+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:02.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:02 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:02.119+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:02.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:02 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:02.119+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:02.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:02 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:02.119+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:02.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:02 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:02.119+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:02.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:02.379+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:02.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:02.379+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:02.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:02.379+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:02.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:02.379+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:02.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:02.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:02.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:02.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:02.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:02.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:02 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:02.309+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:03.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:03.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:03.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:03.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:03.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:03.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:03 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:03.155+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:03 smithi192 podman[72345]: 2024-03-28 18:52:03.406426096 +0000 UTC m=+5.659474605 container died 5c7fd4b54cb09b96405185f776714f6945582d15ea0c485f50fb874460f05f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:03.336+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:03.336+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:03.336+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:03.336+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:03.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:03 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:03.336+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:03.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:03 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:03.307+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:04.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:03 smithi192 podman[72345]: 2024-03-28 18:52:03.80761026 +0000 UTC m=+6.060658818 container cleanup 5c7fd4b54cb09b96405185f776714f6945582d15ea0c485f50fb874460f05f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T18:52:04.092 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:03 smithi192 bash[72345]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8 2024-03-28T18:52:04.358 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72364]: 2024-03-28 18:52:04.092725578 +0000 UTC m=+0.680871630 container remove 5c7fd4b54cb09b96405185f776714f6945582d15ea0c485f50fb874460f05f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:04.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:04.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:04.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:04.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:04.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:04.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:04 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:04.174+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:04.637 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:04.358+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:04.637 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:04.358+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:04.637 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:04.358+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:04.637 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:04.358+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:04.637 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:04 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:04.358+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:04.638 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.286589698 +0000 UTC m=+0.017652876 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:52:04.638 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.401866594 +0000 UTC m=+0.132929797 container create bb096d85d6854efa609ecbfe029ef04379cc587e65777d3beab1afff2866438f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:52:04.638 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.637752726 +0000 UTC m=+0.368815901 container init bb096d85d6854efa609ecbfe029ef04379cc587e65777d3beab1afff2866438f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:04.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:04 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:04.335+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:04.907 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.641572184 +0000 UTC m=+0.372635342 container start bb096d85d6854efa609ecbfe029ef04379cc587e65777d3beab1afff2866438f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T18:52:04.907 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.712535305 +0000 UTC m=+0.443598487 container attach bb096d85d6854efa609ecbfe029ef04379cc587e65777d3beab1afff2866438f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T18:52:05.214 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 28 18:52:04 smithi192 podman[72454]: 2024-03-28 18:52:04.932419035 +0000 UTC m=+0.663482199 container died bb096d85d6854efa609ecbfe029ef04379cc587e65777d3beab1afff2866438f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T18:52:05.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:05.341+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:05.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:05.341+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:05.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:05.341+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:05.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:05.341+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:05.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:05 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:05.341+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:05.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.172+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:05.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T18:52:05.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:05.578 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T18:52:05.578 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T18:52:05.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.9 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:05.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:05 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:05.338+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:05.897 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 systemd[1]: Stopping Ceph osd.9 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:06.276 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.897+0000 7f8d91f41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.897+0000 7f8d91f41700 -1 osd.9 99 *** Got signal Terminated *** 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:05 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:05.897+0000 7f8d91f41700 -1 osd.9 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:06.277 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:06 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:06.143+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:06.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:06.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:06.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:06.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:06.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:06.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:06.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:06.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:06.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:06.351+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:06.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:06 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:06.367+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:07.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:07.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:07.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:07.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:07.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:07.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:07 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:07.096+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:07.599 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:07 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:07.327+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:07.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:07.350+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:07.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:07.350+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:07.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:07.350+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:07.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:07.350+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:07.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:07 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:07.350+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:08.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:08.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:08 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:08.063+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:08.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:08 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:08.359+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:08.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:08 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:08.288+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:09.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:09.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:09.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:09.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:09.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:09.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:09 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:09.066+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:09.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:09 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:09.342+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:09.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:09 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:09.269+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:10.526 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.753168+0000 front 2024-03-28T18:50:53.753150+0000 (oldest deadline 2024-03-28T18:51:16.053274+0000) 2024-03-28T18:52:10.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:01.255130+0000 front 2024-03-28T18:51:01.255144+0000 (oldest deadline 2024-03-28T18:51:27.154950+0000) 2024-03-28T18:52:10.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:10.056429+0000 front 2024-03-28T18:51:10.056205+0000 (oldest deadline 2024-03-28T18:51:33.556247+0000) 2024-03-28T18:52:10.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.856983+0000 front 2024-03-28T18:51:18.857030+0000 (oldest deadline 2024-03-28T18:51:44.157191+0000) 2024-03-28T18:52:10.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:29.359019+0000 front 2024-03-28T18:51:29.358862+0000 (oldest deadline 2024-03-28T18:51:55.258718+0000) 2024-03-28T18:52:10.527 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:10 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9[64190]: 2024-03-28T18:52:10.104+0000 7f8d8a52d700 -1 osd.9 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:36.959898+0000 front 2024-03-28T18:51:36.959916+0000 (oldest deadline 2024-03-28T18:52:02.859732+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:10.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:10.298+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:10.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:10 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:10.308+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:11.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:11 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:11.319+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:11.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:11 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:11.287+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:11.794 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:11 smithi145 podman[91071]: 2024-03-28 18:52:11.53439616 +0000 UTC m=+5.762249090 container died d44955764a8c4a9863e1f0e37e011ade27150424871297f66fc62efc896f4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:52:11.794 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:11 smithi145 podman[91071]: 2024-03-28 18:52:11.778632901 +0000 UTC m=+6.006485824 container cleanup d44955764a8c4a9863e1f0e37e011ade27150424871297f66fc62efc896f4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T18:52:11.794 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:11 smithi145 bash[91071]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9 2024-03-28T18:52:12.367 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91089]: 2024-03-28 18:52:12.063173189 +0000 UTC m=+0.527675847 container remove d44955764a8c4a9863e1f0e37e011ade27150424871297f66fc62efc896f4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:12.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:12 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:12.345+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:12.674 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.270756842 +0000 UTC m=+0.017992696 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:52:12.674 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.380483721 +0000 UTC m=+0.127719550 container create c784f4fb6699a7c2b900c8c06a0872f3140464cb12a7529374f31e847e66ea72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T18:52:12.674 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.583127606 +0000 UTC m=+0.330363436 container init c784f4fb6699a7c2b900c8c06a0872f3140464cb12a7529374f31e847e66ea72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:52:12.674 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.586198243 +0000 UTC m=+0.333434073 container start c784f4fb6699a7c2b900c8c06a0872f3140464cb12a7529374f31e847e66ea72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:52:12.674 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.674765702 +0000 UTC m=+0.422001540 container attach c784f4fb6699a7c2b900c8c06a0872f3140464cb12a7529374f31e847e66ea72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:12.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:12 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:12.261+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:13.026 INFO:journalctl@ceph.osd.9.smithi145.stdout:Mar 28 18:52:12 smithi145 podman[91182]: 2024-03-28 18:52:12.883405422 +0000 UTC m=+0.630641260 container died c784f4fb6699a7c2b900c8c06a0872f3140464cb12a7529374f31e847e66ea72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T18:52:13.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T18:52:13.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:13.509 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T18:52:13.509 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T18:52:13.509 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.10 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:13.538 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.273+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:13.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:13 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:13.363+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:13.812 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 systemd[1]: Stopping Ceph osd.10 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:14.212 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.812+0000 7f94d7a83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:14.212 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.812+0000 7f94d7a83700 -1 osd.10 99 *** Got signal Terminated *** 2024-03-28T18:52:14.212 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:13 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:13.812+0000 7f94d7a83700 -1 osd.10 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:14.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:14.333+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:14.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:14 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:14.292+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:15.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:15 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:15.343+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:15.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:15 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:15.331+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:16.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:16 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:16.305+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:16.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:16.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:16.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:16.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:16.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:16.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:16.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:16 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:16.311+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:17.614 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:17.615 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:17 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:17.344+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:17.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:17.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:17 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:17.263+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:18.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:18 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:18.301+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:18.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:55.091151+0000 front 2024-03-28T18:50:55.091172+0000 (oldest deadline 2024-03-28T18:51:20.991057+0000) 2024-03-28T18:52:18.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.991433+0000 front 2024-03-28T18:51:00.991460+0000 (oldest deadline 2024-03-28T18:51:24.491650+0000) 2024-03-28T18:52:18.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.693205+0000 front 2024-03-28T18:51:09.693160+0000 (oldest deadline 2024-03-28T18:51:37.293834+0000) 2024-03-28T18:52:18.712 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6814 osd.3 since back 2024-03-28T18:51:18.394956+0000 front 2024-03-28T18:51:18.394866+0000 (oldest deadline 2024-03-28T18:51:42.494874+0000) 2024-03-28T18:52:18.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:26.496245+0000 front 2024-03-28T18:51:26.496279+0000 (oldest deadline 2024-03-28T18:51:52.896831+0000) 2024-03-28T18:52:18.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.597963+0000 front 2024-03-28T18:51:37.598004+0000 (oldest deadline 2024-03-28T18:52:01.097982+0000) 2024-03-28T18:52:18.713 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:18 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10[61099]: 2024-03-28T18:52:18.297+0000 7f94d006f700 -1 osd.10 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.600093+0000 front 2024-03-28T18:51:45.600041+0000 (oldest deadline 2024-03-28T18:52:07.899981+0000) 2024-03-28T18:52:19.612 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:19 smithi164 podman[72206]: 2024-03-28 18:52:19.337142058 +0000 UTC m=+5.633421466 container died d193f7bb90fa55a42e690977c0a0de309977304ac609acd175b65e1fbc05534b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:19.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:19 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:19.253+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:19.962 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:19 smithi164 podman[72206]: 2024-03-28 18:52:19.711842632 +0000 UTC m=+6.008122023 container cleanup d193f7bb90fa55a42e690977c0a0de309977304ac609acd175b65e1fbc05534b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T18:52:19.962 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:19 smithi164 bash[72206]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10 2024-03-28T18:52:20.314 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:19 smithi164 podman[72235]: 2024-03-28 18:52:19.963271413 +0000 UTC m=+0.621450686 container remove d193f7bb90fa55a42e690977c0a0de309977304ac609acd175b65e1fbc05534b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T18:52:20.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.217704442 +0000 UTC m=+0.017757488 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:52:20.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.322234404 +0000 UTC m=+0.122287470 container create 3e8590a28a7bb32bddb326c712cffdfb9e455b381fd93b7d232b6d6d2c4e341b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:52:20.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.525266085 +0000 UTC m=+0.325319127 container init 3e8590a28a7bb32bddb326c712cffdfb9e455b381fd93b7d232b6d6d2c4e341b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:20.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.528359601 +0000 UTC m=+0.328412640 container start 3e8590a28a7bb32bddb326c712cffdfb9e455b381fd93b7d232b6d6d2c4e341b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T18:52:20.591 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.591706968 +0000 UTC m=+0.391760006 container attach 3e8590a28a7bb32bddb326c712cffdfb9e455b381fd93b7d232b6d6d2c4e341b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:20.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:20 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:20.281+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:20.962 INFO:journalctl@ceph.osd.10.smithi164.stdout:Mar 28 18:52:20 smithi164 podman[72326]: 2024-03-28 18:52:20.821010513 +0000 UTC m=+0.621063559 container died 3e8590a28a7bb32bddb326c712cffdfb9e455b381fd93b7d232b6d6d2c4e341b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T18:52:21.409 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T18:52:21.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:21.490 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T18:52:21.491 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T18:52:21.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.11 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:21.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.240+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:21.801 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 systemd[1]: Stopping Ceph osd.11 for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:22.157 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.801+0000 7fe056adc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:22.157 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.801+0000 7fe056adc700 -1 osd.11 99 *** Got signal Terminated *** 2024-03-28T18:52:22.157 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:21 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:21.801+0000 7fe056adc700 -1 osd.11 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:52:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:22.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:22.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:22.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:22 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:22.280+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:23.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:23 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:23.321+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:24.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:24.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:24.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:24.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:24.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:24.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:24.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:24.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:24 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:24.310+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-28T18:51:58.679588+0000 front 2024-03-28T18:51:58.679625+0000 (oldest deadline 2024-03-28T18:52:23.979575+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:25.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:25 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:25.328+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-28T18:51:58.679588+0000 front 2024-03-28T18:51:58.679625+0000 (oldest deadline 2024-03-28T18:52:23.979575+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6806 osd.0 since back 2024-03-28T18:50:53.970165+0000 front 2024-03-28T18:50:53.970214+0000 (oldest deadline 2024-03-28T18:51:16.270100+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6804 osd.1 since back 2024-03-28T18:51:00.971405+0000 front 2024-03-28T18:51:00.971292+0000 (oldest deadline 2024-03-28T18:51:25.671154+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-28T18:51:09.772353+0000 front 2024-03-28T18:51:09.772249+0000 (oldest deadline 2024-03-28T18:51:35.072411+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6812 osd.4 since back 2024-03-28T18:51:28.475020+0000 front 2024-03-28T18:51:28.475112+0000 (oldest deadline 2024-03-28T18:51:52.574879+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-28T18:51:37.176325+0000 front 2024-03-28T18:51:37.176367+0000 (oldest deadline 2024-03-28T18:52:03.076393+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.145:6822 osd.6 since back 2024-03-28T18:51:45.877749+0000 front 2024-03-28T18:51:45.877733+0000 (oldest deadline 2024-03-28T18:52:08.177656+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.164:6820 osd.7 since back 2024-03-28T18:51:52.878684+0000 front 2024-03-28T18:51:52.878704+0000 (oldest deadline 2024-03-28T18:52:18.679107+0000) 2024-03-28T18:52:26.657 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:26 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11[61639]: 2024-03-28T18:52:26.316+0000 7fe0530d0700 -1 osd.11 99 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-28T18:51:58.679588+0000 front 2024-03-28T18:51:58.679625+0000 (oldest deadline 2024-03-28T18:52:23.979575+0000) 2024-03-28T18:52:27.710 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:27 smithi192 podman[72570]: 2024-03-28 18:52:27.448980021 +0000 UTC m=+5.761037967 container died a60188623b97afe48e99860a6d7c66d5cb5314942dcf533a289866db85e67228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:28.077 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:27 smithi192 podman[72570]: 2024-03-28 18:52:27.792899145 +0000 UTC m=+6.104957076 container cleanup a60188623b97afe48e99860a6d7c66d5cb5314942dcf533a289866db85e67228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T18:52:28.077 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:27 smithi192 bash[72570]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11 2024-03-28T18:52:28.377 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72587]: 2024-03-28 18:52:28.077242056 +0000 UTC m=+0.631975670 container remove a60188623b97afe48e99860a6d7c66d5cb5314942dcf533a289866db85e67228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T18:52:28.654 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.378150507 +0000 UTC m=+0.101292639 container create 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T18:52:28.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.294591319 +0000 UTC m=+0.017733467 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:52:28.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.580559869 +0000 UTC m=+0.303702007 container init 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-28T18:52:28.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.584685466 +0000 UTC m=+0.307827607 container start 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T18:52:28.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.655466486 +0000 UTC m=+0.378608631 container attach 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:28.907 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:28 smithi192 podman[72677]: 2024-03-28 18:52:28.871327109 +0000 UTC m=+0.594469232 container died 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:52:29.410 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 28 18:52:29 smithi192 podman[72703]: 2024-03-28 18:52:29.408636349 +0000 UTC m=+0.532003812 container remove 5b50cca95889ae372132cbdf8ee125e2645e7ed049d85d857247f34872903891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T18:52:29.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T18:52:29.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:29.512 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T18:52:29.512 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T18:52:29.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.a 2024-03-28T18:52:29.838 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:29 smithi145 systemd[1]: Stopping Ceph mds.a for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:30.277 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:29 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: 2024-03-28T18:52:29.837+0000 7fb34e825700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:30.277 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:29 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a[71024]: 2024-03-28T18:52:29.837+0000 7fb34e825700 -1 mds.a *** got signal Terminated *** 2024-03-28T18:52:33.963 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:33 smithi145 podman[91300]: 2024-03-28 18:52:33.690353473 +0000 UTC m=+3.976977787 container died f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:52:34.276 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:34 smithi145 podman[91300]: 2024-03-28 18:52:34.076675843 +0000 UTC m=+4.363300149 container cleanup f9a899a7e1106b81b0376e4c13dceb0a2ef47706dbd19076d04d7fa09a513532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T18:52:34.276 INFO:journalctl@ceph.mds.a.smithi145.stdout:Mar 28 18:52:34 smithi145 bash[91300]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-a 2024-03-28T18:52:34.487 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T18:52:34.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:34.524 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T18:52:34.524 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T18:52:34.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.d 2024-03-28T18:52:34.776 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:34 smithi145 systemd[1]: Stopping Ceph mds.d for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:35.277 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: 2024-03-28T18:52:34.865+0000 7fd366980700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:35.277 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:34 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d[70065]: 2024-03-28T18:52:34.865+0000 7fd366980700 -1 mds.d *** got signal Terminated *** 2024-03-28T18:52:37.974 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:37 smithi145 podman[91419]: 2024-03-28 18:52:37.673023141 +0000 UTC m=+2.930364286 container died 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:52:37.974 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:37 smithi145 podman[91419]: 2024-03-28 18:52:37.974226984 +0000 UTC m=+3.231568134 container cleanup 20bba0ebfaf68b0571ab1871388926f73fdda5e654cbd381d0bbcb7e3b962395 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T18:52:38.224 INFO:journalctl@ceph.mds.d.smithi145.stdout:Mar 28 18:52:37 smithi145 bash[91419]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-d 2024-03-28T18:52:38.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T18:52:38.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:38.377 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T18:52:38.377 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T18:52:38.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.g 2024-03-28T18:52:38.526 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:38 smithi145 systemd[1]: Stopping Ceph mds.g for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:39.026 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: 2024-03-28T18:52:38.752+0000 7efc30fa1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:39.027 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:38 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g[70673]: 2024-03-28T18:52:38.752+0000 7efc30fa1700 -1 mds.g *** got signal Terminated *** 2024-03-28T18:52:41.929 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:41 smithi145 podman[91540]: 2024-03-28 18:52:41.672873206 +0000 UTC m=+3.070425506 container died bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T18:52:42.236 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 podman[91540]: 2024-03-28 18:52:42.003043879 +0000 UTC m=+3.400596178 container cleanup bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T18:52:42.237 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 bash[91540]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g 2024-03-28T18:52:42.461 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T18:52:42.487 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 podman[91554]: 2024-03-28 18:52:42.237377789 +0000 UTC m=+0.561985078 container remove bf97af9657e4d1b6ad886f7bcb0316cd29ad5a2be71d1cde5a75f69df129c416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-g, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:52:42.487 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.g.service: Deactivated successfully. 2024-03-28T18:52:42.487 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 systemd[1]: Stopped Ceph mds.g for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:52:42.487 INFO:journalctl@ceph.mds.g.smithi145.stdout:Mar 28 18:52:42 smithi145 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.967s CPU time. 2024-03-28T18:52:42.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:42.496 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T18:52:42.496 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T18:52:42.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.j 2024-03-28T18:52:42.776 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:42 smithi145 systemd[1]: Stopping Ceph mds.j for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:43.276 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j[69700]: 2024-03-28T18:52:42.831+0000 7f8c117ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:43.276 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:42 smithi145 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j[69700]: 2024-03-28T18:52:42.831+0000 7f8c117ef700 -1 mds.j *** got signal Terminated *** 2024-03-28T18:52:46.921 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:46 smithi145 podman[91662]: 2024-03-28 18:52:46.591938985 +0000 UTC m=+3.882562862 container died 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True) 2024-03-28T18:52:47.232 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:47 smithi145 podman[91662]: 2024-03-28 18:52:47.073028181 +0000 UTC m=+4.363652063 container cleanup 189b793e27aafe5e07b318a0393ad138d297d739638391ac30ce4bc74e5acb0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T18:52:47.232 INFO:journalctl@ceph.mds.j.smithi145.stdout:Mar 28 18:52:47 smithi145 bash[91662]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-j 2024-03-28T18:52:47.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T18:52:47.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:47.489 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T18:52:47.489 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T18:52:47.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.b 2024-03-28T18:52:47.786 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:47 smithi164 systemd[1]: Stopping Ceph mds.b for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:48.212 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: 2024-03-28T18:52:47.786+0000 7f4366178700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:48.212 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:47 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b[66808]: 2024-03-28T18:52:47.786+0000 7f4366178700 -1 mds.b *** got signal Terminated *** 2024-03-28T18:52:49.962 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:49 smithi164 podman[72435]: 2024-03-28 18:52:49.631517935 +0000 UTC m=+1.952253707 container died d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T18:52:50.270 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 podman[72435]: 2024-03-28 18:52:50.069531998 +0000 UTC m=+2.390267777 container cleanup d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:50.270 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 bash[72435]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b 2024-03-28T18:52:50.503 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T18:52:50.527 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 podman[72449]: 2024-03-28 18:52:50.270814995 +0000 UTC m=+0.631575833 container remove d55e683a26f5c41ea1c6be784d9870137325a9a33c2670d537d1f0e51cf058ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T18:52:50.527 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.b.service: Deactivated successfully. 2024-03-28T18:52:50.527 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 systemd[1]: Stopped Ceph mds.b for b991dbde-ed30-11ee-95d0-87774f69a715. 2024-03-28T18:52:50.528 INFO:journalctl@ceph.mds.b.smithi164.stdout:Mar 28 18:52:50 smithi164 systemd[1]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.b.service: Consumed 4min 35.788s CPU time. 2024-03-28T18:52:50.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:50.540 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T18:52:50.540 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T18:52:50.541 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.e 2024-03-28T18:52:50.856 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:50 smithi164 systemd[1]: Stopping Ceph mds.e for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:51.212 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: 2024-03-28T18:52:50.856+0000 7fa7dfc60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:51.212 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:50 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e[66181]: 2024-03-28T18:52:50.856+0000 7fa7dfc60700 -1 mds.e *** got signal Terminated *** 2024-03-28T18:52:53.915 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:53 smithi164 podman[72554]: 2024-03-28 18:52:53.642040271 +0000 UTC m=+2.887621092 container died 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T18:52:54.212 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:54 smithi164 podman[72554]: 2024-03-28 18:52:54.014705436 +0000 UTC m=+3.260286243 container cleanup 3e7519f502a305bf7fdc3dfa0de0a7e94c0afde870793222d446941390f4c5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T18:52:54.212 INFO:journalctl@ceph.mds.e.smithi164.stdout:Mar 28 18:52:54 smithi164 bash[72554]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-e 2024-03-28T18:52:54.430 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T18:52:54.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:54.466 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T18:52:54.466 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T18:52:54.466 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.h 2024-03-28T18:52:54.712 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:54 smithi164 systemd[1]: Stopping Ceph mds.h for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:55.212 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: 2024-03-28T18:52:54.809+0000 7fcc9522c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:55.212 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:54 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h[66493]: 2024-03-28T18:52:54.809+0000 7fcc9522c700 -1 mds.h *** got signal Terminated *** 2024-03-28T18:52:57.962 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:57 smithi164 podman[72676]: 2024-03-28 18:52:57.631080906 +0000 UTC m=+2.943726505 container died d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T18:52:58.325 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:58 smithi164 podman[72676]: 2024-03-28 18:52:58.043285205 +0000 UTC m=+3.355930800 container cleanup d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:52:58.325 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:58 smithi164 bash[72676]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h 2024-03-28T18:52:58.325 INFO:journalctl@ceph.mds.h.smithi164.stdout:Mar 28 18:52:58 smithi164 podman[72691]: 2024-03-28 18:52:58.211254887 +0000 UTC m=+0.573248280 container remove d3238ae76da1fb486124841732edd16aa5a50773cc100cf7d8b5c96430681e95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T18:52:58.337 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T18:52:58.372 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:52:58.372 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T18:52:58.372 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T18:52:58.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.k 2024-03-28T18:52:58.688 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:52:58 smithi164 systemd[1]: Stopping Ceph mds.k for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:52:58.963 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:52:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: 2024-03-28T18:52:58.688+0000 7f0a5f396700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:52:58.963 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:52:58 smithi164 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k[65862]: 2024-03-28T18:52:58.688+0000 7f0a5f396700 -1 mds.k *** got signal Terminated *** 2024-03-28T18:53:01.962 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:53:01 smithi164 podman[72797]: 2024-03-28 18:53:01.640238421 +0000 UTC m=+3.053622927 container died 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:53:02.372 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:53:02 smithi164 podman[72797]: 2024-03-28 18:53:02.088508246 +0000 UTC m=+3.501892754 container cleanup 04156e0f2135afa91420d2c9a5faf4c553df9e23b8c0a3966f28b6869a89a859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:53:02.372 INFO:journalctl@ceph.mds.k.smithi164.stdout:Mar 28 18:53:02 smithi164 bash[72797]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-k 2024-03-28T18:53:02.514 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T18:53:02.549 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:53:02.549 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T18:53:02.549 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T18:53:02.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.c 2024-03-28T18:53:02.841 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:02 smithi192 systemd[1]: Stopping Ceph mds.c for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:53:03.107 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: 2024-03-28T18:53:02.841+0000 7f1e1bb02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:53:03.107 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:02 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c[66559]: 2024-03-28T18:53:02.841+0000 7f1e1bb02700 -1 mds.c *** got signal Terminated *** 2024-03-28T18:53:05.900 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:05 smithi192 podman[72782]: 2024-03-28 18:53:05.635438356 +0000 UTC m=+2.896089955 container died 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:53:06.157 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:05 smithi192 podman[72782]: 2024-03-28 18:53:05.982628303 +0000 UTC m=+3.243279871 container cleanup 2d0caec4a3c641f372479b7ee10563896ebf0632c67beaf3d52c5d9a95d1e815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:53:06.158 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 28 18:53:05 smithi192 bash[72782]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-c 2024-03-28T18:53:06.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T18:53:06.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:53:06.436 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T18:53:06.436 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T18:53:06.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f 2024-03-28T18:53:06.745 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:06 smithi192 systemd[1]: Stopping Ceph mds.f for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:53:07.157 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[69454]: 2024-03-28T18:53:06.744+0000 7fbb6b0bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:53:07.157 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:06 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f[69454]: 2024-03-28T18:53:06.744+0000 7fbb6b0bb700 -1 mds.f *** got signal Terminated *** 2024-03-28T18:53:09.907 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:09 smithi192 podman[72915]: 2024-03-28 18:53:09.53532099 +0000 UTC m=+2.880706214 container died a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:53:10.286 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:10 smithi192 podman[72915]: 2024-03-28 18:53:10.019839119 +0000 UTC m=+3.365224336 container cleanup a584925d4bbbd69db2cd2e6d97573ae189377883321244e1eb3dbcb624effd27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T18:53:10.287 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 28 18:53:10 smithi192 bash[72915]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-f 2024-03-28T18:53:10.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T18:53:10.485 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:53:10.485 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T18:53:10.486 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T18:53:10.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.i 2024-03-28T18:53:10.807 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:10 smithi192 systemd[1]: Stopping Ceph mds.i for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:53:11.157 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:53:10.806+0000 7f8f37967700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:53:11.157 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:10 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i[66875]: 2024-03-28T18:53:10.806+0000 7f8f37967700 -1 mds.i *** got signal Terminated *** 2024-03-28T18:53:13.899 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:13 smithi192 podman[73037]: 2024-03-28 18:53:13.634613602 +0000 UTC m=+2.932342217 container died 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T18:53:14.158 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:13 smithi192 podman[73037]: 2024-03-28 18:53:13.973353958 +0000 UTC m=+3.271082552 container cleanup 6c1f0d8f7f8212d7bc5f6875fcd5b1650909eb484109c48082342aab26b25885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T18:53:14.158 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 28 18:53:13 smithi192 bash[73037]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-i 2024-03-28T18:53:14.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T18:53:14.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:53:14.419 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T18:53:14.419 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T18:53:14.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.l 2024-03-28T18:53:14.760 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:14 smithi192 systemd[1]: Stopping Ceph mds.l for b991dbde-ed30-11ee-95d0-87774f69a715... 2024-03-28T18:53:15.157 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: 2024-03-28T18:53:14.759+0000 7f0adf119700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:53:15.157 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:14 smithi192 ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l[65938]: 2024-03-28T18:53:14.759+0000 7f0adf119700 -1 mds.l *** got signal Terminated *** 2024-03-28T18:53:17.907 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:17 smithi192 podman[73157]: 2024-03-28 18:53:17.635239195 +0000 UTC m=+3.006267792 container died 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-03-28T18:53:18.356 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:17 smithi192 podman[73157]: 2024-03-28 18:53:17.985207201 +0000 UTC m=+3.356235785 container cleanup 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:53:18.357 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:17 smithi192 bash[73157]: ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l 2024-03-28T18:53:18.357 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 28 18:53:18 smithi192 podman[73172]: 2024-03-28 18:53:18.169437253 +0000 UTC m=+0.530738335 container remove 863eedda497a6eb411c8a648030823e8063eb853cfd6d0b8507d40ff23b06d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-b991dbde-ed30-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T18:53:18.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b991dbde-ed30-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T18:53:18.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:53:18.408 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T18:53:18.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:53:18.599 INFO:teuthology.orchestra.run.smithi145.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:53:21.663 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:53:21.858 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:53:24.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:53:24.699 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T18:53:27.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:53:27.252 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:53:27.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:53:27.314 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T18:53:27.316 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi145/crash 2024-03-28T18:53:27.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:53:27.368 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:53:27.368 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:53:27.370 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi164/crash 2024-03-28T18:53:27.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:53:27.416 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:53:27.416 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:53:27.418 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi192/crash 2024-03-28T18:53:27.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:53:27.457 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:53:27.457 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:53:27.458 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T18:53:27.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T18:53:27.540 INFO:teuthology.orchestra.run.smithi145.stdout:2024-03-28T18:29:49.276005+0000 mon.a (mon.0) 190 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:53:27.540 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T18:53:27.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[SEC\]' /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T18:53:27.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[ERR\]' /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T18:53:27.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[WRN\]' /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T18:53:27.719 INFO:teuthology.orchestra.run.smithi145.stdout:2024-03-28T18:29:49.276005+0000 mon.a (mon.0) 190 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:53:27.720 INFO:tasks.cephadm:Compressing logs... 2024-03-28T18:53:27.720 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:53:27.762 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:53:27.764 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:53:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T18:53:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:53:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T18:53:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T18:53:27.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:53:27.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:53:27.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:53:27.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T18:53:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T18:53:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:53:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T18:53:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:53:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T18:53:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T18:53:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T18:53:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:53:27.819 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T18:53:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T18:53:27.820 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T18:53:27.820 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:53:27.820 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T18:53:27.823 INFO:teuthology.orchestra.run.smithi192.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:53:27.825 INFO:teuthology.orchestra.run.smithi192.stderr: 82.6% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:53:27.826 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T18:53:27.827 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T18:53:27.828 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T18:53:27.828 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.3.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:53:27.830 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:53:27.831 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:53:27.831 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T18:53:27.831 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T18:53:27.831 INFO:teuthology.orchestra.run.smithi145.stderr:gzip/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.9.log: -5 --verbose -- /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T18:53:27.831 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.d.log: /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.a.log: 86.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:53:27.847 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T18:53:27.847 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T18:53:27.883 INFO:teuthology.orchestra.run.smithi145.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T18:53:27.903 INFO:teuthology.orchestra.run.smithi164.stderr: 94.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T18:53:27.921 INFO:teuthology.orchestra.run.smithi192.stderr: 93.2% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T18:53:27.924 INFO:teuthology.orchestra.run.smithi145.stderr: 90.3% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:53:27.934 INFO:teuthology.orchestra.run.smithi192.stderr: 94.5% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T18:53:27.955 INFO:teuthology.orchestra.run.smithi145.stderr: 83.0% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:53:27.967 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:53:27.981 INFO:teuthology.orchestra.run.smithi145.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:53:28.002 INFO:teuthology.orchestra.run.smithi145.stderr: 94.6% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T18:53:28.048 INFO:teuthology.orchestra.run.smithi192.stderr: 86.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:53:28.055 INFO:teuthology.orchestra.run.smithi164.stderr: 82.6% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:53:28.055 INFO:teuthology.orchestra.run.smithi145.stderr: 94.3% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T18:53:28.068 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:53:28.074 INFO:teuthology.orchestra.run.smithi145.stderr: 91.0% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:53:28.133 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:53:28.272 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:53:28.390 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:53:28.464 INFO:teuthology.orchestra.run.smithi164.stderr: 94.6% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T18:53:28.597 INFO:teuthology.orchestra.run.smithi145.stderr: 93.9% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T18:53:29.333 INFO:teuthology.orchestra.run.smithi164.stderr: 91.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T18:53:30.302 INFO:teuthology.orchestra.run.smithi145.stderr: 90.1% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T18:53:31.226 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T18:53:41.120 INFO:teuthology.orchestra.run.smithi145.stderr: 90.3% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T18:54:50.115 INFO:teuthology.orchestra.run.smithi192.stderr: 97.3% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T18:57:38.473 INFO:teuthology.orchestra.run.smithi145.stderr: 97.2% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T18:57:43.731 INFO:teuthology.orchestra.run.smithi145.stderr: 92.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T19:04:25.830 INFO:teuthology.orchestra.run.smithi164.stderr: 92.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T19:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr: 92.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T19:08:25.554 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T19:08:43.334 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T19:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-28T19:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:real 15m15.563s 2024-03-28T19:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:user 6m37.407s 2024-03-28T19:08:43.350 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m18.055s 2024-03-28T19:10:33.729 INFO:teuthology.orchestra.run.smithi192.stderr: 92.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T19:11:19.035 INFO:teuthology.orchestra.run.smithi192.stderr: 92.8% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T19:13:54.321 INFO:teuthology.orchestra.run.smithi145.stderr: 92.6% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T19:15:07.228 INFO:teuthology.orchestra.run.smithi145.stderr: 92.5% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T19:15:46.150 INFO:teuthology.orchestra.run.smithi145.stderr: 92.4% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T19:15:46.153 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-28T19:15:46.153 INFO:teuthology.orchestra.run.smithi145.stderr:real 22m18.379s 2024-03-28T19:15:46.153 INFO:teuthology.orchestra.run.smithi145.stderr:user 8m22.925s 2024-03-28T19:15:46.153 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m24.343s 2024-03-28T19:16:25.092 INFO:teuthology.orchestra.run.smithi192.stderr: 92.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T19:20:48.925 INFO:teuthology.orchestra.run.smithi192.stderr: 92.5% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T19:20:50.443 INFO:teuthology.orchestra.run.smithi192.stderr: 94.7% -- replaced with /var/log/ceph/b991dbde-ed30-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T19:20:50.476 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-28T19:20:50.476 INFO:teuthology.orchestra.run.smithi192.stderr:real 27m22.688s 2024-03-28T19:20:50.476 INFO:teuthology.orchestra.run.smithi192.stderr:user 9m28.476s 2024-03-28T19:20:50.476 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m28.235s 2024-03-28T19:20:50.477 INFO:tasks.cephadm:Archiving logs... 2024-03-28T19:20:50.478 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi145/log 2024-03-28T19:20:50.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T19:22:08.283 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi164/log 2024-03-28T19:22:08.284 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T19:23:11.077 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi192/log 2024-03-28T19:23:11.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T19:24:35.400 INFO:tasks.cephadm:Removing cluster... 2024-03-28T19:24:35.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force 2024-03-28T19:24:35.639 INFO:teuthology.orchestra.run.smithi145.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T19:24:36.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force 2024-03-28T19:24:36.828 INFO:teuthology.orchestra.run.smithi164.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T19:24:37.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b991dbde-ed30-11ee-95d0-87774f69a715 --force 2024-03-28T19:24:38.069 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: b991dbde-ed30-11ee-95d0-87774f69a715 2024-03-28T19:24:38.920 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T19:24:38.920 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T19:24:38.938 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T19:24:38.957 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T19:24:38.972 INFO:tasks.cephadm:Teardown complete 2024-03-28T19:24:38.972 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T19:24:38.983 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T19:24:38.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T19:24:38.985 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T19:24:39.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T19:24:39.200 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T19:24:39.210 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T19:24:39.210 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T19:24:39.213 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T19:24:39.218 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T19:24:39.228 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-28T19:24:39.230 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-28T19:24:39.232 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +575us[ +580us] +/- 78ms 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -1432us[-1427us] +/- 55ms 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +1634us[+1640us] +/- 84ms 2024-03-28T19:24:39.259 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T19:24:39.329 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T19:24:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-28T19:24:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +576us[ +576us] +/- 78ms 2024-03-28T19:24:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1439us[-1437us] +/- 55ms 2024-03-28T19:24:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +1625us[+1625us] +/- 84ms 2024-03-28T19:24:39.330 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T19:24:39.374 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T19:24:39.374 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-28T19:24:39.374 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +555us[ +555us] +/- 78ms 2024-03-28T19:24:39.374 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1457us[-1453us] +/- 55ms 2024-03-28T19:24:39.374 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +1602us[+1602us] +/- 84ms 2024-03-28T19:24:39.375 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T19:24:39.376 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T19:24:39.386 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T19:24:39.386 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T19:24:39.395 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T19:24:39.566 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T19:24:39.685 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T19:24:39.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T19:24:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:39.860 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T19:24:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:39.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T19:24:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:39.925 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T19:24:39.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T19:24:39.947 INFO:teuthology.task.internal:Duration was 3872.628353 seconds 2024-03-28T19:24:39.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T19:24:39.958 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T19:24:39.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T19:24:39.961 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T19:24:39.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T19:24:40.025 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T19:24:40.028 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T19:24:40.029 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T19:24:40.490 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T19:24:40.490 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-03-28T19:24:40.490 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T19:24:40.516 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-03-28T19:24:40.516 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T19:24:40.552 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-03-28T19:24:40.552 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T19:24:40.578 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T19:24:40.578 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T19:24:40.581 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T19:24:40.596 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T19:24:40.661 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T19:24:40.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T19:24:40.664 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T19:24:40.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T19:24:41.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T19:24:41.072 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T19:24:41.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T19:24:41.109 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T19:24:41.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T19:24:41.167 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T19:24:41.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.217 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-03-28T19:24:41.231 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-03-28T19:24:41.248 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-03-28T19:24:41.272 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:41.309 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:41.323 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T19:24:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T19:24:41.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T19:24:41.350 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T19:24:41.351 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi145 2024-03-28T19:24:41.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T19:24:41.403 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi164 2024-03-28T19:24:41.403 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T19:24:41.445 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386/remote/smithi192 2024-03-28T19:24:41.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T19:24:41.487 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T19:24:41.487 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T19:24:41.490 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T19:24:41.493 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T19:24:41.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T19:24:41.549 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T19:24:41.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T19:24:41.560 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T19:24:41.560 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T19:24:41.563 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T19:24:41.565 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T19:24:41.575 INFO:teuthology.orchestra.run.smithi145.stdout: 266033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 19:24 /home/ubuntu/cephtest 2024-03-28T19:24:41.577 INFO:teuthology.orchestra.run.smithi164.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 19:24 /home/ubuntu/cephtest 2024-03-28T19:24:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 19:24 /home/ubuntu/cephtest 2024-03-28T19:24:41.590 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T19:24:41.601 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T19:24:41.624 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T19:24:41.647 DEBUG:teuthology.nuke:shortname: smithi145 2024-03-28T19:24:41.647 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T19:24:41.672 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.002316', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdoIz05eUlhUkEGcz+QQ2ZG7iE+g47sCRa3agonGiyd8To3h80ZtG4I48icaCSjSQNzbVXZ1aJDta5boLBLCD4='} 2024-03-28T19:24:41.676 DEBUG:teuthology.nuke:shortname: smithi164 2024-03-28T19:24:41.676 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T19:24:41.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.003841', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOR0ZjGzjYhg8OQczKBEPQpnNAGAz+5HvUx6kV/hR3LIHpH4bLjAxlPDXfSkxYWgE323Q4hi0BGyLOhGRswGYFc='} 2024-03-28T19:24:41.701 DEBUG:teuthology.nuke:shortname: smithi192 2024-03-28T19:24:41.701 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T19:24:41.729 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-28T19:24:41.730 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T19:24:41.742 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625386', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:11:47.005079', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF9yVmsF8k5npS9xLS17jnLwNvuiscQ44RKd07u+ol1/iBOvFPmrvJBEXon5VQ2EZ3fpMsfbmSnfjkhI4YUIrh4='} 2024-03-28T19:24:41.756 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-28T19:24:41.756 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:41.774 INFO:teuthology.orchestra.console.smithi164:Power off 2024-03-28T19:24:41.774 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T19:24:41.785 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-28T19:24:41.798 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-03-28T19:24:41.798 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:41.812 INFO:teuthology.orchestra.console.smithi192:Power off 2024-03-28T19:24:41.812 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T19:24:41.820 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-28T19:24:41.834 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-03-28T19:24:41.834 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:41.857 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-28T19:24:45.786 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:45.822 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:45.858 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:45.898 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-28T19:24:45.933 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-28T19:24:45.969 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-28T19:24:49.903 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:49.934 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:49.970 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:50.013 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-28T19:24:50.043 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-28T19:24:50.079 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-28T19:24:54.018 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:54.044 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:54.080 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T19:24:54.127 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-28T19:24:54.154 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-03-28T19:24:54.188 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-03-28T19:24:54.228 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-28T19:24:54.254 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-03-28T19:24:54.289 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-03-28T19:24:54.394 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} duration: 3872.628353357315 failure_reason: '"2024-03-28T18:29:49.276005+0000 mon.a (mon.0) 190 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T19:24:54.394 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T19:24:54.485 INFO:teuthology.run:FAIL